:W&Bá¾Þ¦÷•M byeva76uª; 9 /0.28.1+bc2cf24b1bd9fa3a1b6cbeb920087b832afc526f0.70.0”Á0‚‚" 3xw8n7jbntqx byeva76uŠÞ byeva76uescheuller-uc-san-diego spatop-sweep"Ÿ _wandb‚{}  batch_size‚1024  dropout‚0.22872469857419828  hidden_dim‚64 ( learning_rate‚0.00011212701630929652  num_embedding_layers‚10Bdpfx_blocks_byeva76ubox8or7c7j$dp-spatop-sweepagent-blocks-v2-fb276Š Õ¬õÒÀﳬš3 (H¨¸hp"3.12.10*0.28.1j linux-x86_64óg ; byeva76u*  _wandb‚ {"runtime":0}  _runtime‚0Ÿ ’è_ÒÙ -Linux-6.8.0-124-generic-x86_64-with-glibc2.36CPython 3.12.10 Õ¬õÒ‰©w*-of*>/data/spatop/options_files/sweep_fixed_dp/config_byeva76u.json*-l* /scratch/logs*-n*dpfx_blocks_byeva76u2-m spanet.trainRdanielprimosch@gmail.comZ /scratch/logsb$dp-spatop-sweepagent-blocks-v2-fb276r/opt/conda/bin/pythonº nruacy5o7sy4msgkmsuvdya1nbs4nwcdæPò2  requirements.txtÛ:oW byeva76uZF (H¨¸ (H¨¸ (hpð"3.12.10*0.28.1j linux-x86_64"ÏÎY byeva76uZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.28.1j linux-x86_64k* byeva76ub trainer/global_step2+š/\  byeva76uZK (H¨¸ (H¨¸(8hpð"3.12.10*0.28.1j linux-x86_64nÎÁ…/  byeva76ub*"trainer/global_step2ØénaK  byeva76uj: Õ¬õÒؼóÍ*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] ØSåÇ ÒÁ€@ˆ@’NVIDIA A100 80GB PCIe˜¢ /€€€€ý/€àú½$ª€Àû”ÌÂS NVIDIA A100 80GB PCIe€€€€À€6"Ampere*(GPU-ed01d6ad-b673-30ac-3501-763022903663Ê13.2º nruacy5o7sy4msgkmsuvdya1nbs4nwcdéXâ’  byeva76uj€  Ö¬õÒ¨÷†>î â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 220 K │ train │ 0 │ │ 1 │ encoder │ JetEncoderWithPairwise │ 332 K │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 4.8 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 12 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ ¬ŠU~[ byeva76ujÉ Ö¬õÒ ìÐ>·Trainable params: 5.3 M Non-trainable params: 50 Total params: 5.3 M Total estimated model params size (MB): 21.374 Modules in train mode: 2326 Modules in eval mode: 0 Total FLOPs: 0 `øgp byeva76uj_ Ö¬õÒÙæ>Pwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). [Ïyz byeva76uji Ö¬õÒÇì>Zwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). -Å«·p byeva76uj_ Ö¬õÒÈÒð>Pwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). y%ÒÝs byeva76ujb Ö¬õÒðô>Swandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). [øñ–m byeva76uj\ Ö¬õÒ¨ž÷>Mwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). Oc‘+ byeva76u*™  transformer_dim‚64  transformer_dim_scale‚2.0  initial_embedding_dim‚64  position_embedding_dim‚64  num_encoder_layers‚4 ! num_branch_embedding_layers‚3  num_branch_encoder_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1  num_attention_heads‚8 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚1  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚"all"  pairwise_block_embeddings‚1  pairwise_cross_type‚true  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 K event_info_file‚7"/data/spatop/event_files/v11/tt_hadronic_v7_full.yaml" F training_file‚4"/data/spatop/tpm70_wpm30_FB350_fixed/all_merged.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_limit‚1.0  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 epochs‚50 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"¬.L ê byeva76ujØ Ö¬õÒà¾î@È/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. FæóÐM byeva76uj» Ö¬õÒØðú@«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. 0ë byeva76ujÙ ׬õÒ¨ÚÁðÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. âhNíP byeva76uj¾ ׬õÒØ…Øð­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. ÛhT;‚(:± ä¬õÒøó°À disk.nvme0c0n1.in‚ 4.35546875 disk.nvme0n1.in‚ 4.35546875 network.sent‚37669% disk./.usageGB‚1167.4886779785156 disk.dm-1.out‚ 0.2333984375 disk.nvme1c1n1.in‚0 disk.md0.in‚0 disk.md0.out‚ 0.58203125 disk.dm-0.out‚24.97802734375# disk.nvme0n1.out‚24.43798828125" disk.nvme1c1n1.out‚ 1.998046875 disk.nvme1n1.out‚ 1.998046875$ memory_percent‚98.33567142486572 proc.memory.rssMB‚ 2797.1875 cpu‚27.202059084256486 network.recv‚41728( disk./.usagePercent‚76.0455090687846 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1n1.in‚0( proc.memory.availableMB‚ 272.68359375* proc.memory.percent‚17.072677612304688 disk.dm-0.in‚ 4.35546875% disk.nvme0c0n1.out‚24.43798828125 proc.cpu.threads‚17Ü%ª‚(:… ä¬õÒó®éÀ gpu.0.gpu‚58.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚44.0 gpu.0.powerWatts‚146.253' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚48.751 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õݨ3b/ lr-AdamW/pg2"trainer/global_step2:JPÚo[G3b/ lr-AdamW/pg1"trainer/global_step2:JPá`-) _wandbruntime‚19  _runtime‚19'œ]½ ' lr-AdamW/pg2‚6.8592057417672025e-06 trainer/global_step‚49 % _timestamp‚1.7845018658842082e+09 _runtime‚ 19.948645896 _step‚0 ' lr-AdamW/pg1‚6.8592057417672025e-06ãâû€Ä¿ ' lr-AdamW/pg1‚6.8592057417672025e-06 ' lr-AdamW/pg2‚6.8592057417672025e-06 trainer/global_step‚49 % _timestamp‚1.7845018658842082e+09 _runtime‚ 19.948645896 _step‚0Æ¥÷n bj GDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated"trainer/global_step2:JPÞ ¨W!bS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JPë”möq"bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JPˆ×p#bl IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JP– Òn$bj GDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JPn͉æn%bj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JP— æi&be BDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current"trainer/global_step2:JP—Š[g'bc @DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed"trainer/global_step2:JPfRoAh(bd ADeviceStatsMonitor.on_train_batch_start/oversize_segments.current"trainer/global_step2:JP¸a´Pl)bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JPSîün*bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current"trainer/global_step2:JP.|®h+bd ADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed"trainer/global_step2:JPËëïs,bo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JP!÷?\p-bl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JPm&_Gk.bg DDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated"trainer/global_step2:JPF¿ªl/bh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JP\¹¡e0ba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPÊDeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JPlj9¹b3b^ ;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak"trainer/global_step2:JP©i²àg4bc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JP“°mzl5bh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JPË"Râr6bn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JPti´°e7ba >DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JP´xEm8bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JPhÅ ´i9be BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JPb úc:b_ bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPõ«™¥q?bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JPï³Ožn@bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JPX!¢ïnAbj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JPßwõnBbj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JP’Í?hCbd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JPÚWË“rDbn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JP¥DÞeeEba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JP¢°½…kFbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JP_LBqGbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JPŽštïmHbi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JP­€ÛcIb_ DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JP9Ug°lRbh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JP™{¼g]SbY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JP(˜pTbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JPË/ nUbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JP/o­úfVbb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JP©|,vWbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JP7¿ÏÌbXb^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JPbò€ÆpYbl IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated"trainer/global_step2:JPA#¸qZbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JPöÔ£ij[bf CDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed"trainer/global_step2:JPjÝét\bp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPÁ‚ªÅa]b] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JPu«Ý{o^bk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JPÿ$Z7m_bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JP>"–6o`bk HDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JP7ÕÎjabf CDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated"trainer/global_step2:JPôZPobbk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JPíxÉdxcbt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPìÊ5jdbf CDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated"trainer/global_step2:JPŠY>Ëhebd ADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed"trainer/global_step2:JP  0mjfbf CDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current"trainer/global_step2:JP1ŽMrggbc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JPœMþ%ghbc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JP\âÂCdib` =DeviceStatsMonitor.on_train_batch_start/segment.all.allocated"trainer/global_step2:JP_V Vfjbb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JPRsB ikbe BDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak"trainer/global_step2:JP>zûqlbm JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JP5%‹jmbf CDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current"trainer/global_step2:JPÜŒ¼snbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPY"kobg DDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated"trainer/global_step2:JPÔ/öfpbb ?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak"trainer/global_step2:JPJZŽÔaqb] :DeviceStatsMonitor.on_train_batch_start/active.all.current"trainer/global_step2:JPCýÿŒgrbc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JPÔ5 nsbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JP ¯`tb\ 9DeviceStatsMonitor.on_train_batch_start/segment.all.freed"trainer/global_step2:JPX€ö@oubk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JPUU¡Ánvbj GDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JP}`¼lwbh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JP½¹÷„ixbe BDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak"trainer/global_step2:JP£IĆrybn KDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JPµ¼Þáczb_ DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 1911930151424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11965 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 34210602496 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚124053 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 1912035004416 GDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 GDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11894 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712Ôð z'ôN NCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚503866 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 34210602496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 1801293222912 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 1886952644246 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159787520 K@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚873905 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚370037 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 1836998977024 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 1852866905712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚691 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 1877719548928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313356800 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11894 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚216687 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 1911930151424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11965 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 34210602496 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚124053 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 1912035004416 GDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 GDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11894 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚494130 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 193204736 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 1877736588288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 NCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚370037 ]MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 35512549376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 1852849866352 _step‚1 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218108416™;„cožbj GDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current"trainer/global_step2:JP/DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JP›áT­o£bj GDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JPüÊm¤bh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JP#¹…Œp¥bk HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated"trainer/global_step2:JPœFÓ¬e¦b` =DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak"trainer/global_step2:JPâö)Ê^§bY 6DeviceStatsMonitor.on_train_batch_end/active.all.freed"trainer/global_step2:JP ¦gîi¨bd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JPk°¯ua©b\ 9DeviceStatsMonitor.on_train_batch_end/segment.all.current"trainer/global_step2:JP(ºñøfªba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JP„œ±gm«bh EDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current"trainer/global_step2:JP<%ÑDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JPøu‰Œl­bg DDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed"trainer/global_step2:JPâ5Äu®bp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JPÿtÜ7g¯bb ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JPÜ“Vh°bc @DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak"trainer/global_step2:JPŠß7s±bn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPœŸkg²bb ?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current"trainer/global_step2:JP\òØim³bh EDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current"trainer/global_step2:JP„†0Ôe´b` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JPbu‘rµbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPf’ö.k¶bf CDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated"trainer/global_step2:JP»Â®»m·bh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JP«µ“[b¸b] :DeviceStatsMonitor.on_train_batch_end/allocation.all.freed"trainer/global_step2:JP ˜ÐÅl¹bg DDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak"trainer/global_step2:JPè°jºbe BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JPõ%<e»b` =DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed"trainer/global_step2:JP’\p“h¼bc @DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current"trainer/global_step2:JPz÷¡Èl½bg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JP–þh¾bc @DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current"trainer/global_step2:JP;¾>>f¿ba >DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPä»–wiÀbd ADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated"trainer/global_step2:JPªÙdÑgÁbb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JP’ø°ÌmÂbh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JP¯!bfÃba >DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JPbªPdÄb_ DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP¼r¢ˆqÓbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JP¥m…Ü_ÔbZ 7DeviceStatsMonitor.on_train_batch_end/segment.all.freed"trainer/global_step2:JPpÔaïbÕb] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JP`T]ÖbX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JPÚ<¤ƒm×bh EDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated"trainer/global_step2:JP’Ê jØbe BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JPî(öqÙbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JPÔàÜÖuÚbp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JPÍ”åhÛbc @DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current"trainer/global_step2:JP Ð_ÜbZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JP³cönÝbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JPÒ©üÀeÞb` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JPeäñ4dßb_ oëbj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JPónkìbf CDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JPŸì1Êoíbj GDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current"trainer/global_step2:JPÛo“ˆdîb_ DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JP¤á)£oðbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JP6Üëmñbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JP­´Hªdòb_ /Q¨sóbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP¶X\2pôbk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JP˜ª]mcõb^ ;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated"trainer/global_step2:JPr¨ÍÇpöbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JPÚ‹ú#n÷bi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JPçÎE\øbW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JPé¯Êãbùb] :DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams"trainer/global_step2:JPn™Ñ·eúb` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JP(²Ï hûbc @DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated"trainer/global_step2:JPŸ L„iübd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JPï% eýb` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JPþ·9ðjþbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JPµ­5=rÿbm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JPP{n€bi FDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JP˜$ã]bX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JP;r‚bm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JP6ÀnÖnƒbi FDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JPcžÙj„be BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JP8¯0Äf…ba >DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JPi¢/Óh†bc @DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak"trainer/global_step2:JPõú³îi‡bd ADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current"trainer/global_step2:JPžé+eˆb` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JPÀâ=…V‰bQ .DeviceStatsMonitor.on_train_batch_end/num_ooms"trainer/global_step2:JP‚ð%ýmŠbh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed"trainer/global_step2:JPh‰ÌIk‹bf CDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current"trainer/global_step2:JP>‘ÔõqŒbl IDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated"trainer/global_step2:JPì³›òrbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JPJö-ŽpŽbk HDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current"trainer/global_step2:JP?õlbg DDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak"trainer/global_step2:JP‘±‹wbr ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPy°- m‘bh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JPz·žYl’bg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JPþÔj“be BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated"trainer/global_step2:JP,Rr”bm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated"trainer/global_step2:JPZ…öm•bh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JP.œXw–br ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPâÇü€`—b[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JPÇ1 /g˜bb ?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed"trainer/global_step2:JPù¯Í.™) _wandbruntime‚20  _runtime‚20ŽnKîKšúL VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚891138 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 _runtime‚ 20.202081932 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚377358 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚126532 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 1889738973472 _step‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚456 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 1836988380160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 34976532992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 36392331776 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 1924500663450 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 34976532992 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚220974 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 1873380711936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 A6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚881386 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚504028 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚347254 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1873181485056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 1836963476480 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚49 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚691 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193843712 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 1924603849532 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 E:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚881386 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚504028 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚377360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚513778 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 N6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 1950079488512 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 % _timestamp‚1.7845018661387548e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 1949974628352 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 34888712192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚891138 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚377360 MBDeviceStatsMonitor.on_train_batch_end/inactive_séš~÷5plit.all.allocated‚347509 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚220977 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174323200 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 1889721934112 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160(þ¬„&›þL M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 LADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚377360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚513778 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 N6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 1950079488512 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 % _timestamp‚1.7845018661387548e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 1949974628352 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 34888712192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚891138 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚377360 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚347509 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚220977 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174323200 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 1889721934112 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚891138 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 _runtime‚ 20.202081932 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚377358 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚126532 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 1889738973472 _step‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚456 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 1836988380160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 34976532992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 36392331776 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 1924500663450 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 34976532992 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚220974 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 1873380711936 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 A6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚881386 LADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚504028 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚347254 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1873181485056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 1836963476480 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚49 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚691 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193843712 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 1924603849532 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 E:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚881386 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚504028 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152kø ¦Aœb< loss/FBt2/assignment_loss"trainer/global_step2:JPP¦“@b; loss/FRt2/detection_loss"trainer/global_step2:JP@nß @žb; loss/FRt1/detection_loss"trainer/global_step2:JPÕÏ¿W-Ÿb( epoch"trainer/global_step2:JPh)9á7 b2 loss/total_loss"trainer/global_step2:JPœÎVC¡b> loss/SRqqt1/assignment_loss"trainer/global_step2:JPïÛC¢b> loss/SRqqt2/assignment_loss"trainer/global_step2:JP{ 1A£b< loss/FBt1/assignment_loss"trainer/global_step2:JPùiA5B¤b= loss/SRqqt1/detection_loss"trainer/global_step2:JP9Íû@¥b; loss/FBt2/detection_loss"trainer/global_step2:JPÝ&’A¦b< loss/FRt2/assignment_loss"trainer/global_step2:JPæï—%B§b= loss/SRqqt2/detection_loss"trainer/global_step2:JP;ÕS@¨b; loss/FBt1/detection_loss"trainer/global_step2:JPÈ€â|A©b< loss/FRt1/assignment_loss"trainer/global_step2:JPXb1¸.ª) _wandbruntime‚20  _runtime‚20€ Æõ«ï .loss/FBt2/detection_loss‚12.82359083717068 _step‚3 /loss/FRt2/assignment_loss‚5.168072873175738 1loss/SRqqt2/detection_loss‚2.3055322425865503 .loss/FBt1/detection_loss‚35.64522032439709 trainer/global_step‚49 /loss/FRt1/assignment_loss‚4.776365594221995 0loss/FBt2/assignment_loss‚2.2298327843348185 .loss/FRt2/detection_loss‚2.188657308387202 .loss/FRt1/detection_loss‚1.369185830394809 epoch‚0 % _timestamp‚1.7845018661397624e+09 _runtime‚ 20.203009714 %loss/total_loss‚79.83858458307932 0loss/SRqqt1/assignment_loss‚4.19383210672749 1loss/SRqqt2/assignment_loss‚4.254707209626387 /loss/FBt1/assignment_loss‚2.235416923250471 1loss/SRqqt1/detection_loss‚2.6481705488060876¹8ù¬ó 0loss/FBt2/assignment_loss‚2.2298327843348185 .loss/FRt2/detection_loss‚2.188657308387202 .loss/FRt1/detection_loss‚1.369185830394809 epoch‚0 % _timestamp‚1.7845018661397624e+09 _runtime‚ 20.203009714 %loss/total_loss‚79.83858458307932 0loss/SRqqt1/assignment_loss‚4.19383210672749 1loss/SRqqt2/assignment_loss‚4.254707209626387 /loss/FBt1/assignment_loss‚2.235416923250471 1loss/SRqqt1/detection_loss‚2.6481705488060876 .loss/FBt2/detection_loss‚12.82359083717068 _step‚3 /loss/FRt2/assignment_loss‚5.168072873175738 1loss/SRqqt2/detection_loss‚2.3055322425865503 .loss/FBt1/detection_loss‚35.64522032439709 trainer/global_step‚49 /loss/FRt1/assignment_loss‚4.776365594221995;cÝ;­‚(:° ó¬õÒéËõ¿' proc.memory.availableMB‚ 272.2265625( proc.memory.percent‚17.0729398727417 cpu‚26.748030729580396" disk.nvme1n1.out‚ 2.39306640625 network.sent‚47292 network.recv‚47918% disk./.usageGB‚1167.4886589050293 disk.rbd0.out‚0% disk.nvme0c0n1.out‚25.65380859375 disk.nvme0n1.in‚ 15.953125 disk.nvme1c1n1.in‚0 disk.rbd0.in‚0$ disk.nvme1c1n1.out‚ 2.39306640625 disk.md0.out‚0.796875 disk.dm-0.out‚ 26.013671875 disk.dm-1.in‚0 disk.dm-1.out‚ 0.3310546875# proc.memory.rssMB‚ 2797.23046875 disk.md0.in‚0$ memory_percent‚98.33846092224121 proc.cpu.threads‚17) disk./.usagePercent‚76.04550782641455 disk.nvme0c0n1.in‚ 15.953125# disk.nvme0n1.out‚25.65380859375 disk.nvme1n1.in‚0 disk.dm-0.in‚ 15.953125I Ý›®‚(: ó¬õÒ²°À gpu.0.gpu‚48.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚44.0 gpu.0.powerWatts‚110.029' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.67633333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Àv/.¯) _wandbruntime‚38  _runtime‚38aÊ¥Cð½ ' lr-AdamW/pg2‚1.3858395274182716e-05 trainer/global_step‚99 % _timestamp‚1.7845018843862724e+09 _runtime‚ 38.451814834 _step‚4 ' lr-AdamW/pg1‚1.3858395274182716e-05,[^DZÁ ' lr-AdamW/pg1‚1.3858395274182716e-05 ' lr-AdamW/pg2‚1.3858395274182716e-05 trainer/global_step‚99 % _timestamp‚1.7845018843862724e+09 _runtime‚ 38.451814834 _step‚4–hJ.²) _wandbruntime‚38  _runtime‚38“Ø{'³õN ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚679371 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚464 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201331200 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 67932253302 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3746054949888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚990882 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 68236157952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3696470293712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1726969 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3586026181632 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 3764385507654 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3656514605056 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 68018399912 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1000618 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 3814291107840 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 3814186254848 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 68148344320 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1000618 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚736087 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3746037910528 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚247929 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 _step‚5 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3764488693624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚736089 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚990882 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 3814291107840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚464 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201331200 XHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 67932253302 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 3746054949888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 NCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚990882 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 68236157952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3696470293712 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1726969 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 3586026181632 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 3764385507654 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 3656514605056 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 68018399912 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1000618 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 3814291107840 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 3814186254848 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 68148344320 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1000618 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 J?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚736087 TGDeviceStatsMonitorÀ¹6º .on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 3746037910528 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚247929 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 _step‚5 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 3764488693624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 RGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚736089 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚990882 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 3814291107840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚685875 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 3783404277760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209712640 _step‚6 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚743408 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 3692904406528 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 71367876608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193843712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184808960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 3852230731776 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚435474 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 68694351468 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 3783404277760 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1000780 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚435477 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 _runtime‚ 38.708619099 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 68826454016 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚686131 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3621746242560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 3802036712940 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 % _timestamp‚1.7845018846452184e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚464 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1753940 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚250654 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 trainer/global_step‚99 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1753940 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1000780 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3733342361472 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 3783421317120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3621721338880 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 68914274816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1010530 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 68608204746 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1744188 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 NDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712acŒ&·†M VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 68608204746 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1744188 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 NDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1010530 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 3852335591936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 3852335591936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚743410 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 3693114119168 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚685875 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 3783404277760 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209712640 _step‚6 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚743408 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 3692904406528 _ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 71367876608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193843712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184808960 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 3852230731776 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚435474 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 68694351468 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 3783404277760 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1000780 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚435477 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 _runtime‚ 38.708619099 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 68826454016 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚686131 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 3621746242560 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 3802036712940 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 % _timestamp‚1.7845018846452184e+09 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚464 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚1753940 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚250654 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 trainer/global_step‚99 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1753940 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1000780 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3733342361472 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 3783421317120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 3621721338880 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ZJDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 68914274816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1010530èÓÊ.¸) _wandbruntime‚38  _runtime‚38Ré@÷¹ñ epoch‚0 _step‚7 1loss/SRqqt1/assignment_loss‚4.192820111007402 /loss/FRt2/detection_loss‚1.9431650688096602 1loss/SRqqt2/detection_loss‚2.4456291614619436 .loss/FBt1/detection_loss‚25.03107426906454 $loss/total_loss‚67.3843276452809 % _timestamp‚1.7845018846462307e+09 /loss/FRt1/assignment_loss‚4.632778605568075 /loss/FRt2/assignment_loss‚5.365819492695494 1loss/SRqqt2/assignment_loss‚4.215025801150525 .loss/FBt2/detection_loss‚11.68248434085399 trainer/global_step‚99 0loss/FBt2/assignment_loss‚2.1980202607810497 _runtime‚ 38.709135329 /loss/FBt1/assignment_loss‚2.184722772960005 /loss/FRt1/detection_loss‚0.9370584806740702 1loss/SRqqt1/detection_loss‚2.5557292802541545 ÔÃûºõ /loss/FRt1/assignment_loss‚4.632778605568075 /loss/FRt2/assignment_loss‚5.365819492695494 1loss/SRqqt2/assignment_loss‚4.215025801150525 .loss/FBt2/detection_loss‚11.68248434085399 trainer/global_step‚99 0loss/FBt2/assignment_loss‚2.1980202607810497 _runtime‚ 38.709135329 /loss/FBt1/assignment_loss‚2.184722772960005 /loss/FRt1/detection_loss‚0.9370584806740702 1loss/SRqqt1/detection_loss‚2.5557292802541545 epoch‚0 _step‚7 1loss/SRqqt1/assignment_loss‚4.192820111007402 /loss/FRt2/detection_loss‚1.9431650688096602 1loss/SRqqt2/detection_loss‚2.4456291614619436 .loss/FBt1/detection_loss‚25.03107426906454 $loss/total_loss‚67.3843276452809 % _timestamp‚1.7845018846462307e+09yÑQ7»‚(:¬ ‚­õÒÂÕö¿ disk.rbd0.out‚0 disk.nvme1n1.in‚0 disk.dm-0.in‚19.53125 disk.dm-1.in‚0 disk.nvme1c1n1.in‚0 disk.dm-0.out‚ 35.3232421875) proc.memory.percent‚17.07296371459961 cpu‚26.53263752086204 network.sent‚93969 disk.nvme0c0n1.in‚19.53125 disk.nvme0n1.in‚19.53125" disk.nvme0n1.out‚ 34.6533203125 disk.md0.out‚ 0.96484375 disk.dm-1.out‚ 0.58740234375$ memory_percent‚98.34203720092773& proc.memory.availableMB‚ 271.640625) disk./.usagePercent‚76.04551652300495$ disk.nvme0c0n1.out‚ 34.6533203125$ disk.nvme1c1n1.out‚ 3.35498046875" disk.nvme1n1.out‚ 3.35498046875 disk.md0.in‚0! proc.memory.rssMB‚ 2797.234375 proc.cpu.threads‚17 network.recv‚52717% disk./.usageGB‚1167.4887924194336 disk.rbd0.in‚0hüŽJ¼‚(:… ‚­õÒîâÂÀ gpu.0.gpu‚64.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚203.409' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚67.803 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0L¿.½) _wandbruntime‚56  _runtime‚56¹6…ÙÊ¾Ä trainer/global_step‚149 % _timestamp‚1.7845019028352697e+09 _runtime‚56.900821035999996 _step‚8 ' lr-AdamW/pg1‚2.0857584806598228e-05 ' lr-AdamW/pg2‚2.0857584806598228e-05åƒ}NÎ¿È % _timestamp‚1.7845019028352697e+09 _runtime‚56.900821035999996 _step‚8 ' lr-AdamW/pg1‚2.0857584806598228e-05 ' lr-AdamW/pg2‚2.0857584806598228e-05 trainer/global_step‚149¾ªAN.À) _wandbruntime‚56  _runtime‚56ÒÌýÇÁ‡O MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 5716547211264 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 102173899776 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1017346 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1102139 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 5476410561536 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2589771 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2599509 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1102139 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 105416797696 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 5370784044032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 102086086144 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1497370 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚371927 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 105601613824 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5716442358272 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 102173899776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 5614356272128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193843712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184816128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0ƒ{3?‘'‹O D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚2589771 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚2599509 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1102139 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 105416797696 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 5370784044032 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 102086086144 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1497370 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚371927 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 105601613824 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5716442358272 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 102173899776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 5614356272128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 193843712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184816128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 5716547211264 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 102173899776 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1017346 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1102139 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 5476410561536 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 \Rßé.Ã) _wandbruntime‚57  _runtime‚57ðÊ6˜&Ä’M EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1024093 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193843712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1507282 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 5512805045760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 102852016640 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚649977 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚149 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1507282 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 _step‚10 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 102852016640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5679366390266 % _timestamp‚1.7845019030921845e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 5406504104960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2616742 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1497532 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5651739678720 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚374382 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 5512597430272 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1024359 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182711808 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1109458 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5406479201280 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 106300940800 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5679469576348 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 57.155466854 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 5651722639360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1109458 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 5651722639360 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚649974 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚699 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚374119 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 106118228992 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 102437680154 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330,÷=wÀ!Å–M VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 102852016640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5679366390266 % _timestamp‚1.7845019030921845e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 5406504104960 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚2616742 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1497532 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 5651739678720 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚374382 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 5512597430272 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1024359 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182711808 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1109458 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 5406479201280 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 106300940800 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5679469576348 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 57.155466854 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 5651722639360 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1109458 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 5651722639360 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚649974 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚699 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚374119 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 106118228992 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 102437680154 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1024093 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 193843712 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.o†™ú~Ün_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚1507282 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 5512805045760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 102852016640 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚649977 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚149 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1507282 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 _step‚10 ãÑ‚©.Æ) _wandbruntime‚57  _runtime‚57Å’àüÇö 1loss/SRqqt1/assignment_loss‚4.191541314125061 /loss/FRt1/detection_loss‚0.6124991637312788 0loss/SRqqt2/detection_loss‚2.279794522702462 %loss/total_loss‚69.93693225029374 epoch‚0 trainer/global_step‚149 /loss/FRt1/assignment_loss‚4.548016341442757 1loss/SRqqt2/assignment_loss‚4.204163261888556 /loss/FBt2/detection_loss‚10.018556156313755 1loss/SRqqt1/detection_loss‚2.9865176918867387 /loss/FBt1/detection_loss‚29.843269399056833 % _timestamp‚1.7845019030930727e+09 _step‚11 /loss/FRt2/assignment_loss‚5.331174980994709 0loss/FBt1/assignment_loss‚2.1109555115302405 0loss/FBt2/assignment_loss‚2.1222875538006636 /loss/FRt2/detection_loss‚1.6881563528206762 _runtime‚ 57.155904775ÈM#TÈú 1loss/SRqqt1/assignment_loss‚4.191541314125061 /loss/FRt1/detection_loss‚0.6124991637312788 0loss/SRqqt2/detection_loss‚2.279794522702462 %loss/total_loss‚69.93693225029374 epoch‚0 trainer/global_step‚149 /loss/FRt1/assignment_loss‚4.548016341442757 1loss/SRqqt2/assignment_loss‚4.204163261888556 /loss/FBt2/detection_loss‚10.018556156313755 1loss/SRqqt1/detection_loss‚2.9865176918867387 /loss/FBt1/detection_loss‚29.843269399056833 % _timestamp‚1.7845019030930727e+09 _step‚11 /loss/FRt2/assignment_loss‚5.331174980994709 0loss/FBt1/assignment_loss‚2.1109555115302405 0loss/FBt2/assignment_loss‚2.1222875538006636 /loss/FRt2/detection_loss‚1.6881563528206762 _runtime‚ 57.155904775 m•»=É‚(:² ‘­õÒýÅÄ¿ disk.rbd0.out‚0) proc.memory.percent‚17.07296371459961 network.sent‚142046% disk./.usageGB‚1167.4888534545898 disk.md0.in‚0 disk.dm-0.out‚36.80908203125 disk.nvme0c0n1.in‚19.53125% disk.nvme0c0n1.out‚36.43896484375$ memory_percent‚98.34532737731934' proc.memory.availableMB‚ 271.1015625 disk.nvme0n1.in‚19.53125# disk.nvme0n1.out‚36.43896484375 disk.nvme1c1n1.in‚0 disk.dm-1.out‚ 0.61865234375 disk.rbd0.in‚0 proc.cpu.threads‚17 network.recv‚57621) disk./.usagePercent‚76.04552049858914$ disk.nvme1c1n1.out‚ 3.89697265625" disk.nvme1n1.out‚ 3.89697265625 disk.md0.out‚ 1.33203125 disk.dm-1.in‚0! proc.memory.rssMB‚ 2797.234375 cpu‚26.401419884670126 disk.nvme1n1.in‚0 disk.dm-0.in‚19.53125ÅÇê…Ê‚(:’ ‘­õÒ½õ’À gpu.0.gpu‚90.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚172.392' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚57.464000000000006 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 /ºÛ5Ë‚(:ª  ­õÒ¼Âß¿$ disk./.usageGB‚1167.855484008789# disk.nvme1c1n1.out‚ 6.3349609375 disk.dm-0.in‚19.53125 proc.cpu.threads‚17 disk.nvme1c1n1.in‚0 disk.md0.out‚ 1.9140625 disk.dm-1.in‚0 disk.rbd0.in‚0# memory_percent‚98.3454704284668& proc.memory.availableMB‚ 271.078125 network.sent‚189766 network.recv‚62363 disk.nvme0n1.in‚19.53125 disk.nvme1n1.in‚0! disk.dm-0.out‚127.50439453125 disk.dm-1.out‚ 1.02099609375 disk.rbd0.out‚0! proc.memory.rssMB‚ 2797.234375" disk.nvme0n1.out‚ 126.326171875! disk.nvme1n1.out‚ 6.3349609375 disk.md0.in‚0 disk.nvme0c0n1.in‚19.53125$ disk.nvme0c0n1.out‚ 126.326171875) proc.memory.percent‚17.07296371459961 cpu‚26.316835557149258( disk./.usagePercent‚76.0694013358599Z+£›Ì‚(:  ­õÒö°ŽÀ gpu.0.gpu‚30.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚201.412' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.13733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÷'.Í) _wandbruntime‚75  _runtime‚75øšf²Äξ _runtime‚ 75.391786248 _step‚12 ' lr-AdamW/pg1‚2.7856774339013742e-05 ' lr-AdamW/pg2‚2.7856774339013742e-05 trainer/global_step‚199 $ _timestamp‚1.784501921327288e+09Ý&ä«ÈÏ trainer/global_step‚199 $ _timestamp‚1.784501921327288e+09 _runtime‚ 75.391786248 _step‚12 ' lr-AdamW/pg1‚2.7856774339013742e-05 ' lr-AdamW/pg2‚2.7856774339013742e-05 [á€B.Ð) _wandbruntime‚75  _runtime‚75ëO'чO VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚860187 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 7296089201152 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚496648 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 7155566810112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7519251234470 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7383660030352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1468189 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7618698461696 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3462311 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7519354420440 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178524672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1994122 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7383677069712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚473 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 140343866368 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 _runtime‚ 75.392273049 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1984386 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚199 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7618698461696 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚860184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7482691673088 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 136023827968 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1468187 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 $ _timestamp‚1.784501921329255e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7482691673088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1356835 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚708 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 7155541906432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1994122 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 7618803314688 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚496381 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 7295885772800 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 135677350728 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 _step‚13 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 136111641600 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7618803314688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7482674633728 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1468187 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 136111641600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚496648 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 7155566810112 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 7519251234470 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7383660030352 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1468189 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7618698461696 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚3462311 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 7519354420440 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178524672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1994122 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7383677069712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚473 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 140343866368 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 _runtime‚ 75.392273049 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚1984386 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚199 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7618698461696 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚860184 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 7482691673088 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 136023827968 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1468187 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 $ _timestamp‚1.784501921329255e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 7482691673088 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1356835 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚708 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 7155541906432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1994122 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚270 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 7618803314688 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚496381 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 7295885772800 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 135677350728 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 _step‚13 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 136111641600 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7618803314688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 7482674633728 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1468187 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 136111641600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ΟK}.Ó) _wandbruntime‚75  _runtime‚75æÌ e¬Ô’M F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3479544 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1475510 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1475508 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 NDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 7656742938624 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1363368 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7520058040320 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 7332483222528 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 % _timestamp‚1.7845019215851789e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209712640 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 136789758464 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7332273509888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚864477 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 136267155562 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7520041000960 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 141221255168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1475508 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1994284 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7520041000960 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7420532098112 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7656847798784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7556799253674 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 136701937664 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3479544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 MADeviceStatsMonitor.on_train_batch_end/allocati– zÊì on.small_pool.freed‚1994284 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚708 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2004034 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7420549137472 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1475510 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚473 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 136353302284 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 136789758464 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 141036446208 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1363638 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¶Þ]Ôœ&Õ–M D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 7656847798784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 7556799253674 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 136701937664 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3479544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1994284 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚708 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2004034 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 7420549137472 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1475510 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚473 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 136353302284 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 136789758464 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 141036446208 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1363638 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3479544 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1475510 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1475508 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 NDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 7656742938624 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1363368 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 7520058040320 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 7332483222528 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 % _timestamp‚1.7845019215851789e+09 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209712640 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 136789758464 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7332273509888 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚864477 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 136267155562 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 7520041000960 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 141221255168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1475508 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚1994284 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 7520041000960 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7420532098112 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 9520954565120 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2481138 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9115554708480 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 9521059418112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 169961569792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8940324672512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 9521059418112 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1696706 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 170049383424 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1074687 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 $ _timestamp‚1.784501939850299e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 9351010034688 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚473 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 9396787283848 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 169506826136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 9350992995328 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 170049383424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220205568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 175254939648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1834237 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚708 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1074684 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 9227263418352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8940299768832 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1834239]~€¸àO YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 169961569792 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 9520954565120 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2481138 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9115554708480 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 9521059418112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 169961569792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8940324672512 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 9521059418112 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚1696706 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚277 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 170049383424 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1074687 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 $ _timestamp‚1.784501939850299e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 9351010034688 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚473 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 9396787283848 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 169506826136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 9350992995328 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 170049383424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 220205568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 175254939648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1834237 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚708 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1074684 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 9227263418352 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚280 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 8940299768832 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 LDDeviceStatsMonitor.on_train_batch_sta-·éÉÛrt/segment.small_pool.allocated‚152 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1834239 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1834239 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2481138 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 9227280457712 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 9520954565120 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 9350992995328 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2490874 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2490874 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 9351010034688 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚622022 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 195301888 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚249 _runtime‚ 93.913351782 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚4315375 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 _step‚17 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚4325113 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1834237 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚622299 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9115774914048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚1696986 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360ª‘è9.á) _wandbruntime‚94  _runtime‚94ý‚š&â”M aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8976019829760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚249 N8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 175954863616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 9264135486112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 9388359362560 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2491036 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4342346 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2491036 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8975994926080 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚624775 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 9434232117082 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1841560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4332594 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1841558 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 9559103902208 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 % _timestamp‚1.7845019401049075e+09 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1841558 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚708 _step‚18 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 9434335303164 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1078974 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 170096630970 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 9388376401920 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 170639679488 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 170182777692 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 170727500288 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1703499 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2500786 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4332594 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1703752 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1078977 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚624525 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 170639679488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1841560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 170727500288 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 176139672576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184808960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 9558999042048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9152159502336 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _runtime‚ 94.168322394 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4342346 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209712640 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 9559103902208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482a={ž&ã˜M A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1841560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4332594 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1841558 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 9559103902208 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 % _timestamp‚1.7845019401049075e+09 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1841558 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚708 _step‚18 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 9434335303164 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1078974 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 170096630970 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 9388376401920 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 170639679488 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 170182777692 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 170727500288 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1703499 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2500786 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4332594 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1703752 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1078977 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚624525 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 170639679488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1841560 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 170727500288 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 176139672576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184808960 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 9558999042048 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9152159502336 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _runtime‚ 94.168322394 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4342346 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209712640 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 9559103902208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 8976019829760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚249 N8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 175954863616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 9264135486112 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 9388359362560 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2491036 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚4342346 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2491036 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8975994926080 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚624775 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 9434232117082 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816µy.ä) _wandbruntime‚94  _runtime‚94}8Œüøåò _step‚19 /loss/FRt1/assignment_loss‚4.529561422966622 0loss/SRqqt2/assignment_loss‚4.11676590574726 0loss/FBt1/assignment_loss‚1.4755078764522778 .loss/FBt2/detection_loss‚8.171811947330044 trainer/global_step‚249 /loss/FRt2/detection_loss‚1.4440393690601923 0loss/SRqqt2/detection_loss‚1.356997166160201 %loss/total_loss‚77.29972384589931 /loss/FRt2/assignment_loss‚5.324730809777975 1loss/SRqqt1/assignment_loss‚4.151685981523423 % _timestamp‚1.7845019401059697e+09 0loss/FBt2/assignment_loss‚1.1672394045761654 /loss/FRt1/detection_loss‚0.4271533989089516 0loss/SRqqt1/detection_loss‚4.215098403482919 .loss/FBt1/detection_loss‚40.91913215991329 epoch‚0 _runtime‚ 94.168947315ðBÉüæö /loss/FRt2/assignment_loss‚5.324730809777975 1loss/SRqqt1/assignment_loss‚4.151685981523423 % _timestamp‚1.7845019401059697e+09 0loss/FBt2/assignment_loss‚1.1672394045761654 /loss/FRt1/detection_loss‚0.4271533989089516 0loss/SRqqt1/detection_loss‚4.215098403482919 .loss/FBt1/detection_loss‚40.91913215991329 epoch‚0 _runtime‚ 94.168947315 _step‚19 /loss/FRt1/assignment_loss‚4.529561422966622 0loss/SRqqt2/assignment_loss‚4.11676590574726 0loss/FBt1/assignment_loss‚1.4755078764522778 .loss/FBt2/detection_loss‚8.171811947330044 trainer/global_step‚249 /loss/FRt2/detection_loss‚1.4440393690601923 0loss/SRqqt2/detection_loss‚1.356997166160201 %loss/total_loss‚77.29972384589931ËÝú7ç‚(:¬ ¾­õÒ Ãç¿' proc.memory.availableMB‚ 207.7734375 network.recv‚70657) disk./.usagePercent‚76.06523045109842 disk.rbd0.out‚0 disk.nvme0n1.in‚19.53125 disk.dm-0.in‚19.53125 disk.dm-0.out‚ 334.048828125 disk.dm-1.in‚0 disk.dm-1.out‚ 1.2314453125 network.sent‚240153 disk.nvme0c0n1.in‚19.53125 disk.nvme1c1n1.in‚0$ memory_percent‚98.73185157775879! proc.memory.rssMB‚ 2797.234375 proc.cpu.threads‚17 disk.nvme1n1.in‚0% disk./.usageGB‚1167.7914505004883 disk.rbd0.in‚0% disk.nvme0c0n1.out‚273.1201171875# disk.nvme0n1.out‚273.1201171875) proc.memory.percent‚17.07296371459961 cpu‚26.209674533115205# disk.nvme1c1n1.out‚ 68.271484375! disk.nvme1n1.out‚ 68.271484375 disk.md0.in‚0 disk.md0.out‚2.828125o"ºè‚(:… ¾­õÒŦ×À gpu.0.gpu‚74.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚112.896' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.632 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08èÁ0é+ _wandbruntime‚112  _runtime‚112ÃÉÃÄê¾ & lr-AdamW/pg1‚4.185515340384477e-05 & lr-AdamW/pg2‚4.185515340384477e-05 trainer/global_step‚299 % _timestamp‚1.7845019583663898e+09 _runtime‚ 112.431602128 _step‚20išêÈë & lr-AdamW/pg1‚4.185515340384477e-05 & lr-AdamW/pg2‚4.185515340384477e-05 trainer/global_step‚299 % _timestamp‚1.7845019583663898e+09 _runtime‚ 112.431602128 _step‚20ˆÈ-0ì+ _wandbruntime‚112  _runtime‚112_†ˆí›O QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚5187915 % _timestamp‚1.7845019583669448e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2200287 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2987626 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚473 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 _step‚21 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚10725057631232 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2200287 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚11274116961286 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 203899311616 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚10935446897664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207622656 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚11274220147256 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚11423315521536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2977890 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2200289 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 210364362752 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1289184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 203987125248 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11423210668544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.bï’peak‚ 743477248 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2200289 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚11423315521536 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 trainer/global_step‚299 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚11219311356928 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚747013 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2036197 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5178177 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2987626 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 203899311616 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11219311356928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚11423210668544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 203987125248 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1289187 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11219328396288 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚10935239275008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 210181643776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2036462 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚5178177 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2987626 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 203899311616 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚11219311356928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚11423210668544 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 203987125248 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1289187 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚11219328396288 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚10935239275008 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 210181643776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2036462 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1289184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 203987125248 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11423210668544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2200289 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚11423315521536 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482'Õ¦¤0ï+ _wandbruntime‚112  _runtime‚112î÷q9ª&ð¤M I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2997538 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5205148 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2207610 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2042943 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 203926106378 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚11461360005632 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2207608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10760752788480 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5195396 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 204577421312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213906944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189003264 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11107738874112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚11256694763520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚708 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1293474 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 204665242112 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 204577421312 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2997538 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2987788 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚473 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 204665242112 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2043205 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5205148 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2987788 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚11311768166572 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚11256694763520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚11311664980490 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 204012253100 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 _runtime‚ 112.685406739 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784501958622114e+09 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5195396 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152Ä®rEñ¨M B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚5195396 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 204577421312 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213906944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 189003264 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚11107738874112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚11256694763520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚708 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1293474 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 204665242112 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 204577421312 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚2997538 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2987788 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚473 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 204665242112 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2043205 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚5205148 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2987788 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚11311768166572 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚11256694763520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚11311664980490 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 204012253100 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 _runtime‚ 112.685406739 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784501958622114e+09 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚5195396 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2997538 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚5205148 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2207610 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2042943 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 203926106378 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚11461360005632 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2207608 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚10760752788480 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0( Eo0ò+ _wandbruntime‚112  _runtime‚112ñ1n0úóô .loss/FBt2/detection_loss‚5.261830488157229 &loss/total_loss‚59.241686072644214 _runtime‚ 112.68587237 /loss/FRt1/assignment_loss‚4.561517692687341 /loss/FBt1/assignment_loss‚0.768830536554257 0loss/SRqqt2/detection_loss‚1.230982662909721 trainer/global_step‚299 _step‚23 1loss/SRqqt2/assignment_loss‚3.344023628205788 /loss/FRt2/detection_loss‚1.4489354277556796 0loss/SRqqt1/detection_loss‚4.792772696529691 $ _timestamp‚1.784501958623016e+09 epoch‚0 /loss/FRt2/assignment_loss‚5.298578325307594 2loss/SRqqt1/assignment_loss‚3.4316846857199796 0loss/FBt2/assignment_loss‚0.3067470531662305 /loss/FRt1/detection_loss‚0.4476120885546796 /loss/FBt1/detection_loss‚28.348170787096024ÒPþôø epoch‚0 /loss/FRt2/assignment_loss‚5.298578325307594 2loss/SRqqt1/assignment_loss‚3.4316846857199796 0loss/FBt2/assignment_loss‚0.3067470531662305 /loss/FRt1/detection_loss‚0.4476120885546796 /loss/FBt1/detection_loss‚28.348170787096024 .loss/FBt2/detection_loss‚5.261830488157229 &loss/total_loss‚59.241686072644214 _runtime‚ 112.68587237 /loss/FRt1/assignment_loss‚4.561517692687341 /loss/FBt1/assignment_loss‚0.768830536554257 0loss/SRqqt2/detection_loss‚1.230982662909721 trainer/global_step‚299 _step‚23 1loss/SRqqt2/assignment_loss‚3.344023628205788 /loss/FRt2/detection_loss‚1.4489354277556796 0loss/SRqqt1/detection_loss‚4.792772696529691 $ _timestamp‚1.784501958623016e+09¬XŸAõ‚(:¶ Í­õÒ¢»ê¿& disk.nvme0c0n1.out‚274.56005859375 disk.nvme1c1n1.in‚0 disk.nvme1n1.in‚0# disk.nvme1n1.out‚68.91064453125 disk.md0.in‚0 disk.dm-0.out‚ 335.142578125) proc.memory.percent‚17.07296371459961 network.sent‚300097( disk./.usagePercent‚76.0652386507408$ disk./.usageGB‚1167.791576385498 disk.rbd0.in‚0 disk.nvme0n1.in‚ 21.6640625$ disk.nvme0n1.out‚274.56005859375 disk.md0.out‚ 3.22265625 disk.dm-0.in‚ 21.6640625 network.recv‚74241 disk.dm-1.out‚ 1.3681640625 disk.nvme0c0n1.in‚ 21.6640625' proc.memory.availableMB‚ 207.4140625! proc.memory.rssMB‚ 2797.234375 cpu‚26.1731510497178 proc.cpu.threads‚17 disk.rbd0.out‚0% disk.nvme1c1n1.out‚68.91064453125$ memory_percent‚98.73404502868652 disk.dm-1.in‚0Pù N›ö‚(: Í­õÒš”•À gpu.0.gpu‚39.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚139.561' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚46.52033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KXë0÷+ _wandbruntime‚130  _runtime‚130†h¸Ãø½ & lr-AdamW/pg1‚4.885434293626028e-05 & lr-AdamW/pg2‚4.885434293626028e-05 trainer/global_step‚349 % _timestamp‚1.7845019768560169e+09 _runtime‚ 130.91916251 _step‚24ùG ;ÇùÁ & lr-AdamW/pg1‚4.885434293626028e-05 & lr-AdamW/pg2‚4.885434293626028e-05 trainer/global_step‚349 % _timestamp‚1.7845019768560169e+09 _runtime‚ 130.91916251 _step‚24Q/-ð0ú+ _wandbruntime‚130  _runtime‚130×ÕCA 'ûšO TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚13325466771968 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 237837053440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6050717 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 237924867072 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚13151653010664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚710 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2566339 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3474642 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 237165776952 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚475 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2566337 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13087646757888 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚13087629718528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚12754932327424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2376016 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 $ _timestamp‚1.784501976856867e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚13087629718528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚12914470194352 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚13325571624960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚13325466771968 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3474642 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184816128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6040979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12509815493632 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3484378 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚13325571624960 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 237924867072 _step‚25èAˤ'üžO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2566337 trainer/global_step‚349 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚13325466771968 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3474642 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184816128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6040979 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚12509815493632 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3484378 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚13325571624960 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 237924867072 _step‚25 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚13325466771968 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 237837053440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6050717 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 237924867072 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚13151653010664 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚710 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2566339 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3474642 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 237165776952 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚475 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2566337 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚13087646757888 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚13087629718528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚12754932327424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2376016 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 $ _timestamp‚1.784501976856867e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚13087629718528 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚12914470194352 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚13325571624960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6058198 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚475 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚13125013125120 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213906944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚12951359301472 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚12951342262112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3494290 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 238515163136 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚874810 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12791533950976 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 238602983936 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚26 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2573660 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2382784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 245809393152 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚710 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 237841728508 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 238602983936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3484540 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7845019771106107e+09 _runtime‚ 131.173904079 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 J>DeviceStatsMonitorê?½BØ .on_train_batch_end/allocation.all.allocated‚6067950 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 245998396416 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6058198 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12545535554560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6067950 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚13363616109056 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2383041 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1507974 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12545510650880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3484540 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12791320044032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3494290 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104š¯&ÿ©M D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3484540 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚12791320044032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3494290 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚13125013125120 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚13189097843898 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚13124996085760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6058198 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚475 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚13125013125120 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213906944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚12951359301472 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚12951342262112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3494290 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 238515163136 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚874810 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12791533950976 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 238602983936 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚26 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2573660 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2382784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 245809393152 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚710 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 237841728508 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 238602983936 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3484540 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7845019771106107e+09 _runtime‚ 131.173904079 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6067950 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 245998396416 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6058198 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚12545535554560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6067950 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚13363616109056 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2383041 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1507974 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚12545510650880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 Fƒ‚(:³ Ü­õҪȖÀ" disk.nvme1n1.out‚ 69.9833984375 network.recv‚78640# disk.nvme0c0n1.out‚ 298.02734375 disk.md0.in‚0 disk.dm-1.in‚0) proc.memory.percent‚17.07296371459961! disk.nvme0n1.out‚ 298.02734375$ disk.nvme1c1n1.out‚ 69.9833984375 disk.nvme1n1.in‚0 disk.dm-0.in‚ 32.13671875 disk.dm-0.out‚ 358.560546875! proc.memory.rssMB‚ 2797.234375 cpu‚26.14721328623838 disk.nvme1c1n1.in‚0) disk./.usagePercent‚76.06529505434146% disk./.usageGB‚1167.7924423217773 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 32.13671875 disk.nvme0n1.in‚ 32.13671875 disk.dm-1.out‚ 1.5791015625 network.sent‚345999 disk.md0.out‚ 3.65234375# memory_percent‚98.7368106842041' proc.memory.availableMB‚ 206.9609375 proc.cpu.threads‚17„‚ç„‚(:’ Ü­õÒûùÅÀ gpu.0.gpu‚77.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.471' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.157000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TŸ²Y0…+ _wandbruntime‚149  _runtime‚149 ±[PƆÀ ' lr-AdamW/pg1‚5.5853532468675795e-05 ' lr-AdamW/pg2‚5.5853532468675795e-05 trainer/global_step‚399 % _timestamp‚1.7845019952798223e+09 _runtime‚ 149.342955191 _step‚28Ì#,Ê‡Ä ' lr-AdamW/pg1‚5.5853532468675795e-05 ' lr-AdamW/pg2‚5.5853532468675795e-05 trainer/global_step‚399 % _timestamp‚1.7845019952798223e+09 _runtime‚ 149.342955191 _step‚28Nèt³0ˆ+ _wandbruntime‚149  _runtime‚149 ¤5&¡'‰›O K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3971394 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218108416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚15227722875392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3981130 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚14294573356032 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 270909105750 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _step‚29 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14758090621712 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 271774795264 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14955948080128 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚14574806192640 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15028982688102 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6903781 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 271774795264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 trainer/global_step‚399 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14758073582352 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _runtime‚ 149.344918194 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚997093 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚14955965119488 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15227722875392 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14955965119488 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3981130 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2715532 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚14574588084224 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 270995252360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6913519 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 193204736 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482õ±£üˆŠŸO JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 280014728192 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15227722875392 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚14955965119488 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3981130 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2715532 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚14574588084224 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 270995252360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6913519 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 193204736 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3971394 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218108416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚15227722875392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3981130 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚14294573356032 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 270909105750 UGDeviceStatsMon<®öñitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _step‚29 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14758090621712 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 271774795264 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚14955948080128 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚14574806192640 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15028982688102 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6903781 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 271774795264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 trainer/global_step‚399 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚14758073582352 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _runtime‚ 149.344918194 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚997093 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚14955965119488 H6°«&Œ¥M QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 272540725760 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 272452904960 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚999556 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚14611192869888 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14330268513280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2722288 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1722477 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186906112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 272540725760 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2939708 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _step‚30 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚15066633893388 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2939710 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚14993314447360 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3991042 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚14993331486720 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚14610981060096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3981292 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 280712546816 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 271585057194 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2939708 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211809792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3991042 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3981292 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2722030 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6930752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15265767352320 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14993331486720 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6921000 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14794962689472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6921000 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2¥ËçT¯&©M H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2939708 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211809792 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3991042 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3981292 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2722030 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6930752 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15265767352320 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚14993331486720 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6921000 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚14794962689472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6921000 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 272540725760 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 272452904960 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚999556 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚14611192869888 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14330268513280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2722288 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1722477 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186906112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 272540725760 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚2939708 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _step‚30 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚15066633893388 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2939710 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚14993314447360 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3991042 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚14993331486720 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚14610981060096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚3981292 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 280712546816 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 271585057194QÕ0Ž+ _wandbruntime‚149  _runtime‚149¿Tû7ÿù 2loss/SRqqt1/assignment_loss‚2.4748601326967945 1loss/FBt2/assignment_loss‚0.32898870108933265 /loss/FRt2/detection_loss‚1.3905792394439682 0loss/SRqqt2/detection_loss‚1.151633403720391 /loss/FBt1/detection_loss‚42.570016745943576 /loss/FRt1/assignment_loss‚4.544661797899188 0loss/FRt1/detection_loss‚0.46198608537652586 1loss/SRqqt1/detection_loss‚3.6625547860873575 epoch‚0 /loss/FBt1/assignment_loss‚0.665317983366549 .loss/FBt2/detection_loss‚6.988318104858567 _runtime‚ 149.602580559 _step‚31 /loss/FRt2/assignment_loss‚5.323240704815503 2loss/SRqqt2/assignment_loss‚2.3841823631105803 %loss/total_loss‚71.94634004840833 trainer/global_step‚399 % _timestamp‚1.7845019955366676e+09o77ý 1loss/SRqqt1/detection_loss‚3.6625547860873575 epoch‚0 /loss/FBt1/assignment_loss‚0.665317983366549 .loss/FBt2/detection_loss‚6.988318104858567 _runtime‚ 149.602580559 _step‚31 /loss/FRt2/assignment_loss‚5.323240704815503 2loss/SRqqt2/assignment_loss‚2.3841823631105803 %loss/total_loss‚71.94634004840833 trainer/global_step‚399 % _timestamp‚1.7845019955366676e+09 2loss/SRqqt1/assignment_loss‚2.4748601326967945 1loss/FBt2/assignment_loss‚0.32898870108933265 /loss/FRt2/detection_loss‚1.3905792394439682 0loss/SRqqt2/detection_loss‚1.151633403720391 /loss/FBt1/detection_loss‚42.570016745943576 /loss/FRt1/assignment_loss‚4.544661797899188 0loss/FRt1/detection_loss‚0.46198608537652586¯fÕÇJ‘‚(:¿ ë­õÒóŠ’À disk.nvme0c0n1.in‚ 33.0078125 disk.nvme0n1.in‚ 33.0078125 disk.md0.in‚0 disk.dm-0.in‚37.0625! disk.dm-0.out‚367.09716796875 disk.rbd0.in‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 206.578125 disk.nvme1n1.in‚ 4.0546875) proc.memory.percent‚17.07296371459961 proc.cpu.threads‚17 network.sent‚393070% disk./.usageGB‚1167.7921028137207$ disk.nvme0n1.out‚306.71337890625 disk.nvme1c1n1.in‚ 4.0546875 disk.nvme1n1.out‚ 70.30859375! proc.memory.rssMB‚ 2797.234375 cpu‚26.127670118423392 network.recv‚83011) disk./.usagePercent‚76.06527294015441& disk.nvme0c0n1.out‚306.71337890625" disk.nvme1c1n1.out‚ 70.30859375 disk.md0.out‚3.828125 disk.dm-1.in‚0 disk.dm-1.out‚ 1.6533203125# memory_percent‚98.7391471862793Yxóᜒ‚(:‘ ë­õÒþ£ºÀ gpu.0.gpu‚88.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.895' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.29833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»QEåG“‚(:¼ ú­õÒþÔú¿ cpu‚26.11144601684846) disk./.usagePercent‚76.06529853297764 disk.nvme0n1.out‚ 310.7421875! proc.memory.rssMB‚ 2797.234375) proc.memory.percent‚17.07296371459961 network.sent‚429129 disk.dm-0.out‚ 370.974609375 disk.dm-1.out‚ 1.79443359375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 33.0078125" disk.nvme0c0n1.out‚ 310.7421875 proc.cpu.threads‚17 disk.nvme1c1n1.in‚ 4.0546875% disk.nvme1c1n1.out‚71.54638671875 disk.nvme1n1.in‚ 4.0546875# disk.nvme1n1.out‚71.54638671875 disk.md0.in‚0 disk.dm-1.in‚0 network.recv‚87694$ disk./.usageGB‚1167.792495727539 disk.md0.out‚4.421875 disk.dm-0.in‚37.0625 disk.nvme0n1.in‚ 33.0078125$ memory_percent‚98.73824119567871' proc.memory.availableMB‚ 206.7265625)¹Í@‘”‚(:† ú­õÒñ®¤À gpu.0.gpu‚91.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚198.354' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚66.118 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þªó0•+ _wandbruntime‚167  _runtime‚167×í=ÁÁ–» % lr-AdamW/pg1‚6.28527220010913e-05 % lr-AdamW/pg2‚6.28527220010913e-05 trainer/global_step‚449 $ _timestamp‚1.784502013702764e+09 _runtime‚ 167.765891232 _step‚32«ÈuZÅ—¿ % lr-AdamW/pg1‚6.28527220010913e-05 % lr-AdamW/pg2‚6.28527220010913e-05 trainer/global_step‚449 $ _timestamp‚1.784502013702764e+09 _runtime‚ 167.765891232 _step‚32 Änl0˜+ _wandbruntime‚167  _runtime‚167Dt´™œO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182718976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚16824283481088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚16824283481088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 305712537088 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3298439 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4468146 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚16079356122112 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 315153313792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 K?DeviceStat•$ª™sMonitor.on_train_batch_start/active.small_pool.freed‚4468146 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚17130083831808 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚16394509435904 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚16394301813248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 $ _timestamp‚1.784502013703571e+09 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 314970594816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 305800350720 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 305712537088 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1932684 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16601694009712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3298437 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207622656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16906518737480 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 304824727768 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 _runtime‚ 167.767763405 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17129978978816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚16824266441728 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16824266441728 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16079331218432 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3298437 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3055003 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3054735 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1932687 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3298439 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16601676970352 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚17129978978816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152.Y®^¦'š O OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4477882 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚7776321 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 304738581158 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚17130083831808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚16906415551510 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 trainer/global_step‚449 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1122051 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 305800350720 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚4477882 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _step‚33 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 315153313792 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4468146 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚17130083831808 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚16394509435904 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚16394301813248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 $ _timestamp‚1.784502013703571e+09 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 314970594816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 305800350720 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 305712537088 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1932684 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16601694009712 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3298437 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207622656 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16906518737480 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 304824727768 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 _runtime‚ 167.767763405 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17129978978816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚16824266441728 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16824266441728 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚16079331218432 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3298437 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3055003 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3054735 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1932687 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3298439 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16601676970352 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚17129978978816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182718976 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚16824283481088 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚16824283481088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 305712537088 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3298439 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4468146 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚16079356122112 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚17168023455744 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16638566077472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3305758 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚16861649848320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3305758 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7793554 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16861632808960 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4478044 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176420352 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚16943963570714 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4487794 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚34 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7793554 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3061541 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 315850375680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16944066756796 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚7783802 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 306390646784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 315673955328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4478044 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4487794 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3305760 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 trainer/global_step‚449 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1124826 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1124567 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚17168128315904 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 305500679324 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482>‚Á̪M I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3305758 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚16861649848320 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3305758 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚7793554 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16861632808960 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4478044 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176420352 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚16943963570714 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4487794 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚34 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚7793554 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3061541 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 315850375680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16944066756796 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚7783802 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 306390646784 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 315673955328 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4478044 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4487794 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3305760 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 trainer/global_step‚449 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1124826 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1124567 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚17168128315904 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 305500679324 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitorE«vuä .on_train_batch_end/segment.all.allocated‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚17168023455744 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16638566077472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104"„Ø+f0ž+ _wandbruntime‚168  _runtime‚168;­ïýŸ÷ %loss/total_loss‚46.82026953607847 epoch‚0 % _timestamp‚1.7845020139589322e+09 _runtime‚ 168.021799402 /loss/FRt1/assignment_loss‚4.570669544258797 1loss/SRqqt2/detection_loss‚1.2395215865653415 2loss/SRqqt2/assignment_loss‚2.2107741431548047 /loss/SRqqt1/detection_loss‚4.88457769029577 trainer/global_step‚449 _step‚35 /loss/FRt2/assignment_loss‚4.934876381055169 0loss/FBt1/assignment_loss‚0.4822364335081407 1loss/FBt2/assignment_loss‚0.26423820009222254 /loss/FRt1/detection_loss‚0.3494031022636945 2loss/SRqqt1/assignment_loss‚2.4845452963466377 .loss/FRt2/detection_loss‚1.381603545812971 /loss/FBt1/detection_loss‚18.762476121740683 .loss/FBt2/detection_loss‚5.255347490984244À_ ] û trainer/global_step‚449 _step‚35 /loss/FRt2/assignment_loss‚4.934876381055169 0loss/FBt1/assignment_loss‚0.4822364335081407 1loss/FBt2/assignment_loss‚0.26423820009222254 /loss/FRt1/detection_loss‚0.3494031022636945 2loss/SRqqt1/assignment_loss‚2.4845452963466377 .loss/FRt2/detection_loss‚1.381603545812971 /loss/FBt1/detection_loss‚18.762476121740683 .loss/FBt2/detection_loss‚5.255347490984244 %loss/total_loss‚46.82026953607847 epoch‚0 % _timestamp‚1.7845020139589322e+09 _runtime‚ 168.021799402 /loss/FRt1/assignment_loss‚4.570669544258797 1loss/SRqqt2/detection_loss‚1.2395215865653415 2loss/SRqqt2/assignment_loss‚2.2107741431548047 /loss/SRqqt1/detection_loss‚4.88457769029577#LBLZM¡‚(: ‰®õÒÚå÷¿ disk.nvme1c1n1.in‚ 4.0546875 disk.nvme1n1.in‚ 4.0546875 disk.dm-0.in‚37.0625 disk.dm-1.in‚0 disk.dm-1.out‚ 1.89208984375 disk.rbd0.in‚0& disk.nvme0c0n1.out‚312.83056640625! proc.memory.rssMB‚ 2797.234375) disk./.usagePercent‚76.06529480586745# disk./.usageGB‚1167.79243850708$ disk.nvme1c1n1.out‚ 71.9814453125 disk.md0.in‚0 disk.rbd0.out‚0 cpu‚26.097685204477493 proc.cpu.threads‚17 network.recv‚91568 disk.nvme0n1.in‚ 33.0078125" disk.nvme1n1.out‚ 71.9814453125' proc.memory.availableMB‚ 206.5390625) proc.memory.percent‚17.07296371459961 network.sent‚445758$ disk.nvme0n1.out‚312.83056640625 disk.md0.out‚4.671875 disk.dm-0.out‚372.8505859375 disk.nvme0c0n1.in‚ 33.0078125# memory_percent‚98.73938560485848K윢‚(:‘ ‰®õÒ—ã¥À gpu.0.gpu‚60.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚116.721' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.907000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù =0£+ _wandbruntime‚186  _runtime‚186EUmĤ¾ trainer/global_step‚499 % _timestamp‚1.7845020321045547e+09 _runtime‚ 186.169608351 _step‚36 & lr-AdamW/pg1‚6.985191153350682e-05 & lr-AdamW/pg2‚6.985191153350682e-05vñȥ & lr-AdamW/pg2‚6.985191153350682e-05 trainer/global_step‚499 % _timestamp‚1.7845020321045547e+09 _runtime‚ 186.169608351 _step‚36 & lr-AdamW/pg1‚6.985191153350682e-05$i)Ñ 0¦+ _wandbruntime‚186  _runtime‚186ÕÆp¢'§œO UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚19032339935232 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189010432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2147184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3664489 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4964898 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚18692584803328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚18692601842688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3664489 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 350100836864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _step‚37 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3664487 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1246822 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚18214000907264 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 $ _timestamp‚1.784502032105133e+09 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚19032235082240 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚18692601842688 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4964898 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3394266 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚17864089080832 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚19032339935232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚18783848414918 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 338654203176 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8639123 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚18692584803328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 339650278912 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8629385 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚184452803583520ï)¦'¨ O UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚18692584803328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 339650278912 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚8629385 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚18445280358352 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚19032339935232 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189010432 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2147184 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚3664489 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4964898 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚18692584803328 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚18692601842688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3664489 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 350100836864 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _step‚37 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3664487 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1246822 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚18214000907264 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 $ _timestamp‚1.784502032105133e+09 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚19032235082240 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚18692601842688 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4964898 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3394266 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚17864089080832 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚19032339935232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚18783848414918 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 338654203176 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚8639123 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2%¹Hå40©+ _wandbruntime‚186  _runtime‚186Ωú ?ª¦M MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚17899809141760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4974796 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 $ _timestamp‚1.784502032359546e+09 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8646604 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 340328388608 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _step‚38 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚18729968209920 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚18250605002752 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3671808 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚18729951170560 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 339244008010 _runtime‚ 186.422892479 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8656356 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 350608954880 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 340328388608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚18482169465472 trainer/global_step‚499 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4984546 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 350795860992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186906112 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 339330154732 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pÞ(¸mool.freed‚18729951170560 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 340416209408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3671810 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1249572 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3400783 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚18482152426112 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚19070384419328 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚19070279559168 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8646604 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3671810 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4974796 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3401049 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚17899784238080 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚18821396434122 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8656356 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4984546 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2151477 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330ŠÌþ °&«ªM S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚19070384419328 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚19070279559168 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8646604 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3671810 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4974796 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3401049 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚17899784238080 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚18821396434122 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8656356 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4984546 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2151477 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚17899809141760 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4974796 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 $ _timestamp‚1.784502032359546e+09 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚8646604 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 340328388608 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _step‚38 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚18729968209920 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚18250605002752 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3671808 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚18729951170560 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 339244008010 _runtime‚ 186.422892479 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8656356 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 350608954880 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 340328388608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚18482169465472 trainer/global_step‚499 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4984546 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 350795860992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186906112 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 339330154732 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚18729951170560 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 340416209408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3671810 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1249572 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3400783 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚18482152426112& h #0¬+ _wandbruntime‚186  _runtime‚186ëõG ü­ö /loss/FRt2/detection_loss‚1.4712301103538403 1loss/SRqqt1/detection_loss‚3.6880460125704606 1loss/SRqqt2/detection_loss‚1.0590990454077394 0loss/FBt2/assignment_loss‚0.1829991874238476 /loss/FRt1/detection_loss‚0.4023089307200113 &loss/total_loss‚61.634154940801004 /loss/FRt2/assignment_loss‚4.603548893364527 /loss/FBt1/detection_loss‚33.251369807496665 .loss/FBt2/detection_loss‚7.439135113032182 _step‚39 /loss/FRt1/assignment_loss‚4.556240410640322 0loss/FBt1/assignment_loss‚0.3704332720721141 epoch‚0 trainer/global_step‚499 % _timestamp‚1.7845020323604279e+09 _runtime‚ 186.42339647 1loss/SRqqt1/assignment_loss‚2.419481482770708 0loss/SRqqt2/assignment_loss‚2.19026267494859ßÙ~®ú 0loss/FBt1/assignment_loss‚0.3704332720721141 epoch‚0 trainer/global_step‚499 % _timestamp‚1.7845020323604279e+09 _runtime‚ 186.42339647 1loss/SRqqt1/assignment_loss‚2.419481482770708 0loss/SRqqt2/assignment_loss‚2.19026267494859 /loss/FRt2/detection_loss‚1.4712301103538403 1loss/SRqqt1/detection_loss‚3.6880460125704606 1loss/SRqqt2/detection_loss‚1.0590990454077394 0loss/FBt2/assignment_loss‚0.1829991874238476 /loss/FRt1/detection_loss‚0.4023089307200113 &loss/total_loss‚61.634154940801004 /loss/FRt2/assignment_loss‚4.603548893364527 /loss/FBt1/detection_loss‚33.251369807496665 .loss/FBt2/detection_loss‚7.439135113032182 _step‚39 /loss/FRt1/assignment_loss‚4.556240410640322'×^ÁßP¯‚(:Å ˜®õÒŸ¸˜À disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 40.42578125 disk.nvme0n1.in‚ 40.42578125 disk.nvme1c1n1.in‚ 4.49609375 disk.md0.out‚5.046875 disk.dm-1.out‚ 2.0576171875! proc.memory.rssMB‚ 2797.234375) proc.memory.percent‚17.07296371459961 cpu‚26.109742692497008 proc.cpu.threads‚17) disk./.usagePercent‚76.06518721662036 disk.nvme1n1.in‚ 4.49609375! disk.dm-0.out‚382.06787109375$ disk.nvme0n1.out‚322.03369140625! disk.nvme1c1n1.out‚ 72.9765625 disk.dm-0.in‚ 44.921875 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚322.03369140625$ memory_percent‚98.74560832977295( proc.memory.availableMB‚ 205.51953125 network.sent‚492888 network.recv‚96537$ disk./.usageGB‚1167.790786743164 disk.nvme1n1.out‚ 72.9765625 disk.md0.in‚0Ñû?¥°‚(:’ ˜®õÒŽùÀÀ gpu.0.gpu‚86.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚117.974' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.324666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïþ0±+ _wandbruntime‚205  _runtime‚205> ü¿Ä²¾ trainer/global_step‚549 % _timestamp‚1.7845020516373086e+09 _runtime‚ 205.700417745 _step‚40 & lr-AdamW/pg1‚7.685110106592233e-05 & lr-AdamW/pg2‚7.685110106592233e-05äõ٨ȳ trainer/global_step‚549 % _timestamp‚1.7845020516373086e+09 _runtime‚ 205.700417745 _step‚40 & lr-AdamW/pg1‚7.685110106592233e-05 & lr-AdamW/pg2‚7.685110106592233e-05(¥!é0´+ _wandbruntime‚205  _runtime‚205b7£'µO [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚20560903164928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚20934491185664 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9501925 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 372483678584 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚20560920204288 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20934491185664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5461650 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5461650 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 % _timestamp‚1.7845020516382475e+09 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚20288883746352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚20934596038656 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213914112 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚19648871846912 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9492187 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _step‚41 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4030537 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚20033896537088 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2361684 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚19648846943232 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚20661281278326 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4030537 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚20288900785712 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5471386 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 trainer/global_step‚549 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚20560920204288 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3733525 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189010432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 373675834368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 373588020736 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3733263 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2361687 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1371579 _runtime‚ 205.701361377 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 373675834368 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4030539 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚20661384464296 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5471386 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 trainer/global_step‚549 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚20560920204288 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3733525 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 189010432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 373675834368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 373588020736 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3733263 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2361687 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1371579 _runtime‚ 205.701361377 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 373675834368 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4030539 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚20661384464296 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20934491185664 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5461650 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5461650 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 % _timestamp‚1.7845020516382475e+09 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚20288883746352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104) <ˆR0·+ _wandbruntime‚205  _runtime‚205íKÙ¢­&¸§M XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 374353951232 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3740342 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 373159630140 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚20972535662592 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2365977 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1374365 % _timestamp‚1.7845020518942955e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3740074 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9519158 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 385717688832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚20698829297530 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5481298 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚19684567004160 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20972640522752 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚20598269532160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 373073483418 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4037858 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9509406 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5471548 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20325772853472 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 374266130432 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5471548 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚20598286571520 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2365977 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1374365 % _timestamp‚1.7845020518942955e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3740074 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9519158 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 385717688832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚20698829297530 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5481298 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚19684567004160 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20972640522752 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚20598269532160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 373073483418 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4037858 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9509406 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5471548 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20325772853472 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 374266130432 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚5471548 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚20598286571520 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 OáO IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22836852142080 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚10364727 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22836747289088 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4396587 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 419900921856 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 419722397184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178524672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 _runtime‚ 224.122913981 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4072063 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22429221526528 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5968138 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚21853530631168 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22538714141734 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚21433604805632 _step‚45 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22836747289088 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4396589 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1496127 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22429238565888 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2576184 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 % _timestamp‚1.7845020700598786e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992A%éì«'Ä¥O N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚22836852142080 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚10364727 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22836747289088 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4396587 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 419900921856 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 419722397184 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178524672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 _runtime‚ 224.122913981 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4072063 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚22429221526528 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5968138 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚21853530631168 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚22538714141734 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚21433604805632 _step‚45 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22836747289088 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4396589 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1496127 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚22429238565888 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2576184 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 % _timestamp‚1.7845020700598786e+09-kÑŸ?0Å+ _wandbruntime‚224  _runtime‚224šI0°&ƪM J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4078798 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22169376241472 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 $ _timestamp‚1.784502070313445e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10381960 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4403908 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182711808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22466587893760 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10372208 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 408291693056 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4403910 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22874896626176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 406902958826 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 408203872256 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1498573 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _step‚46 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 406989105548 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚22466587893760 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 420415828992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5968300 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10381960 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5978050 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2580477 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚21469324866560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22576262160938 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207615488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _runtime‚ 224.376753859 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22466604933120 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22466604933120 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22874896626176 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 trainer/global_step‚599 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4403908 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360-*6–éÇ®M MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4403908 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4078798 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22169376241472 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 $ _timestamp‚1.784502070313445e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚10381960 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4403908 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 769742336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182711808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚22466587893760 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚10372208 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 408291693056 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚4403910 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22874896626176 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 406902958826 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 408203872256 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1498573 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _step‚46 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 406989105548 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚22466587893760 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 420415828992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚596830hy.Ë 0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚10381960 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5978050 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2580477 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚21469324866560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚22576262160938 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207615488 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _runtime‚ 224.376753859 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚22466604933120 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚22466604933120 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22874896626176 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 trainer/global_step‚599.q¦YÕ0È+ _wandbruntime‚224  _runtime‚224YÛmúÉô /loss/FRt1/assignment_loss‚4.543589993824945 -loss/FRt2/assignment_loss‚4.1188638812684 /loss/FBt1/detection_loss‚20.394031193227537 1loss/SRqqt1/assignment_loss‚1.866047847349912 1loss/FBt1/assignment_loss‚0.20525463867647153 0loss/FRt1/detection_loss‚0.35013348027166996 epoch‚0 _step‚47 0loss/SRqqt1/detection_loss‚4.247286151872565 %loss/total_loss‚47.44164321764545 trainer/global_step‚599 % _timestamp‚1.7845020703143482e+09 _runtime‚ 224.37726241 1loss/SRqqt2/assignment_loss‚1.652651108733837 1loss/FBt2/assignment_loss‚0.17462932407273912 .loss/FRt2/detection_loss‚1.355652014923064 1loss/SRqqt2/detection_loss‚1.0873478448118317 -loss/FBt2/detection_loss‚7.44615573861248a¯”×þÊø 1loss/SRqqt2/assignment_loss‚1.652651108733837 1loss/FBt2/assignment_loss‚0.17462932407273912 .loss/FRt2/detection_loss‚1.355652014923064 1loss/SRqqt2/detection_loss‚1.0873478448118317 -loss/FBt2/detection_loss‚7.44615573861248 /loss/FRt1/assignment_loss‚4.543589993824945 -loss/FRt2/assignment_loss‚4.1188638812684 /loss/FBt1/detection_loss‚20.394031193227537 1loss/SRqqt1/assignment_loss‚1.866047847349912 1loss/FBt1/assignment_loss‚0.20525463867647153 0loss/FRt1/detection_loss‚0.35013348027166996 epoch‚0 _step‚47 0loss/SRqqt1/detection_loss‚4.247286151872565 %loss/total_loss‚47.44164321764545 trainer/global_step‚599 % _timestamp‚1.7845020703143482e+09 _runtime‚ 224.37726241/"'r*PË‚(:Å ¶®õÒ±½î¿# disk.nvme1c1n1.out‚ 74.275390625 disk.md0.out‚ 5.58984375 disk.dm-1.out‚ 2.30078125' proc.memory.availableMB‚ 205.0390625% disk./.usageGB‚1167.9159889221191% disk.nvme0c0n1.out‚349.0068359375! disk.nvme1n1.out‚ 74.275390625 disk.rbd0.out‚0! proc.memory.rssMB‚ 2797.234375 proc.cpu.threads‚17 disk.nvme1n1.in‚ 4.49609375 disk.md0.in‚0 disk.dm-0.in‚ 44.921875 disk.dm-1.in‚0 disk.rbd0.in‚0# memory_percent‚98.7485408782959) proc.memory.percent‚17.07296371459961 network.recv‚105238! disk.nvme0c0n1.in‚ 40.42578125 disk.nvme0n1.in‚ 40.42578125 disk.dm-0.out‚408.9072265625 cpu‚26.097091581596512 network.sent‚586909( disk./.usagePercent‚76.0733423821605# disk.nvme0n1.out‚349.0068359375 disk.nvme1c1n1.in‚ 4.49609375Í+b°œÌ‚(:‘ ¶®õÒ§¯œÀ gpu.0.gpu‚19.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.627' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.208999999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬­•íQÍ‚(:Æ Å®õÒßÔÜ¿ disk.nvme0n1.in‚ 40.42578125$ disk.nvme0n1.out‚356.10498046875 disk.md0.in‚0 disk.dm-0.in‚ 44.921875& proc.memory.availableMB‚ 204.984375! proc.memory.rssMB‚ 2797.234375) proc.memory.percent‚17.07296371459961 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.nvme1c1n1.out‚ 74.705078125 disk.nvme1n1.in‚ 4.49609375! disk.nvme1n1.out‚ 74.705078125 disk.md0.out‚ 5.80078125! disk.dm-0.out‚415.86474609375( disk./.usagePercent‚76.0733339340441$ disk./.usageGB‚1167.915859222412 disk.dm-1.in‚0& disk.nvme0c0n1.out‚356.10498046875 disk.nvme1c1n1.in‚ 4.49609375$ memory_percent‚98.74887466430664 network.sent‚623243 network.recv‚110148 disk.dm-1.out‚ 2.40234375! disk.nvme0c0n1.in‚ 40.42578125 cpu‚26.08942138596273tÝv›Î‚(: Å®õҰЉÀ gpu.0.gpu‚66.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚202.913' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.63766666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~¶0Ï+ _wandbruntime‚242  _runtime‚242Ç'>ªÄо & lr-AdamW/pg1‚9.084948013075335e-05 & lr-AdamW/pg2‚9.084948013075335e-05 trainer/global_step‚649 % _timestamp‚1.7845020884927783e+09 _runtime‚ 242.557585313 _step‚48 °_äÈÑ & lr-AdamW/pg1‚9.084948013075335e-05 & lr-AdamW/pg2‚9.084948013075335e-05 trainer/global_step‚649 % _timestamp‚1.7845020884927783e+09 _runtime‚ 242.557585313 _step‚480qÚb}0Ò+ _wandbruntime‚242  _runtime‚242ÉQ §'Ó¡O VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚23672993602048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11217791 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159788544 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 _runtime‚ 242.557644273 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚24416250191112 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 441551318016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182718976 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4762637 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4411437 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1620498 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 440056482790 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 441551318016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4762637 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1620750 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6464890 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23218387571712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 440142629400 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚24297556927488 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚24297539888128 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6455154 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4762639 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11227529 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 454630934016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4411182 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 454813652992 % _timestamp‚1.7845020884933298e+09 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚24739108245504 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚23976090522352 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 trainer/global_step‚649 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 441463504384 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6455154 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24739108245504 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24297556927488 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24739003392512 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207622656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24739003392512 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207622656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 769742336 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 _runtime‚ 242.557644273 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚24416250191112 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 441551318016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182718976 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4762637 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4411437 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1620498 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 440056482790 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 441551318016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4762637 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1620750 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6464890 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23218387571712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 440142629400 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚24297556927488 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚24297539888128 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6455154 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4762639 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11227529 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 454630934016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4411182 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 454813652992 % _timestamp‚1.7845020884933298e+09 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚24739108245504 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚23976090522352 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 trainer/global_step‚649 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 441463504384 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6455154 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24739108245504 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24297556927488 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24777152729600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚24453798210428 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚24012979629472 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6474802 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚24453695024346 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11244762 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11244762 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 J@DeviceStatsMonitor.on_train_batcáˆhYh_end/allocation.small_pool.peak‚11896 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 $ _timestamp‚1.784502088747325e+09 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4769960 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚24777152729600 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2794974 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚23709589235200 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203421184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178517504 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11235010 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 442229434880 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚649 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6474802 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1623237 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚24334923294720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4417951 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 $ _timestamp‚1.784502088747325e+09 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4769960 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚24777152729600 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2794974 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚23709589235200 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203421184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178517504 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11235010 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 442229434880 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚649 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6474802 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1623237 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚24334923294720 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4417951 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24777152729600 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚24453798210428 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚24012979629472 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6474802 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚24453695024346 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11244762 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚22›ºR0Ø+ _wandbruntime‚242  _runtime‚242 xCVýÙ÷ epoch‚0 /loss/FRt1/assignment_loss‚4.018764455390699 1loss/SRqqt1/detection_loss‚3.1823190008608724 $ _timestamp‚1.784502088748239e+09 /loss/FRt2/assignment_loss‚3.859544811196808 1loss/SRqqt1/assignment_loss‚1.978938763392599 1loss/SRqqt2/assignment_loss‚1.602360603195138 0loss/FBt1/assignment_loss‚0.3081257094304557 %loss/total_loss‚41.31412694608055 trainer/global_step‚649 _step‚51 .loss/FBt1/detection_loss‚18.71190862185463 /loss/FBt2/detection_loss‚4.6148434003322025 _runtime‚ 242.811084849 0loss/FBt2/assignment_loss‚0.1662317354617478 0loss/FRt1/detection_loss‚0.33586326172234926 /loss/FRt2/detection_loss‚1.4279744900428064 1loss/SRqqt2/detection_loss‚1.1072520932002368+Y¾Úû /loss/FRt1/assignment_loss‚4.018764455390699 1loss/SRqqt1/detection_loss‚3.1823190008608724 $ _timestamp‚1.784502088748239e+09 /loss/FRt2/assignment_loss‚3.859544811196808 1loss/SRqqt1/assignment_loss‚1.978938763392599 1loss/SRqqt2/assignment_loss‚1.602360603195138 0loss/FBt1/assignment_loss‚0.3081257094304557 %loss/total_loss‚41.31412694608055 trainer/global_step‚649 _step‚51 .loss/FBt1/detection_loss‚18.71190862185463 /loss/FBt2/detection_loss‚4.6148434003322025 _runtime‚ 242.811084849 0loss/FBt2/assignment_loss‚0.1662317354617478 0loss/FRt1/detection_loss‚0.33586326172234926 /loss/FRt2/detection_loss‚1.4279744900428064 1loss/SRqqt2/detection_loss‚1.1072520932002368 epoch‚03¼e¡vPÛ‚(:Å Ô®õÒ–Ü‹À) proc.memory.percent‚17.07296371459961 cpu‚26.08064458733529 network.sent‚639731 disk.dm-0.in‚ 44.921875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 40.42578125 disk.nvme1c1n1.in‚ 4.49609375! proc.memory.rssMB‚ 2797.234375 proc.cpu.threads‚17% disk./.usageGB‚1167.9160346984863" disk.nvme1n1.out‚ 75.7001953125 disk.md0.out‚ 6.37890625% disk.nvme0c0n1.out‚379.0283203125# disk.nvme0n1.out‚379.0283203125) disk./.usagePercent‚76.07334536384865 disk.nvme1n1.in‚ 4.49609375 disk.dm-1.in‚0 disk.dm-1.out‚ 2.7109375 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 40.42578125$ disk.nvme1c1n1.out‚ 75.7001953125$ memory_percent‚98.74956607818604 network.recv‚114219 disk.md0.in‚0 disk.dm-0.out‚ 438.2578125( proc.memory.availableMB‚ 204.87109375©-ošÜ‚(: Ô®õҜ˻À gpu.0.gpu‚63.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚196.34' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.44666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0rÈ'0Ý+ _wandbruntime‚260  _runtime‚260f¼ÇÅÄÞ¾ _runtime‚ 260.998179222 _step‚52 & lr-AdamW/pg1‚9.784866966316887e-05 & lr-AdamW/pg2‚9.784866966316887e-05 trainer/global_step‚699 % _timestamp‚1.7845021069350464e+09c€xöÈß & lr-AdamW/pg2‚9.784866966316887e-05 trainer/global_step‚699 % _timestamp‚1.7845021069350464e+09 _runtime‚ 260.998179222 _step‚52 & lr-AdamW/pg1‚9.784866966316887e-054WD¼®0à+ _wandbruntime‚260  _runtime‚260›”‘@§'á¡O ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 473972104808 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 % _timestamp‚1.7845021069358685e+09 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6951906 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚25003145434112 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 475489059840 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 489740897280 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚26293579868550 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 _runtime‚ 260.999361394 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 475401246208 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 trainer/global_step‚699 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5128687 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5128687 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182718976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚26641259495936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚25819693910352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚26165858249728 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26641259495936 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 475401246208 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4750723 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6951906 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 473885958198 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚25003120530432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚25819710949712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4751000 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3005187 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6961642 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚26165875289088 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12090331 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 489558178304 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚26641364348928 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1745539 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚25492886331392 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚26165875289088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207622656®wJŽíâ¥O G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚26165875289088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207622656 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 473972104808 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311680887 % _timestamp‚1.7845021069358685e+09 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6951906 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚25003145434112 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 475489059840 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 489740897280 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚26293579868550 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 _runtime‚ 260.999361394 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 475401246208 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86146610 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 trainer/global_step‚699 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5128687 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5128687 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182718976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚26641259495936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚25819693910352 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚26165858249728 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚26641259495936 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 475401246208 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4750723 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚6951906 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚274 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 473885958198 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚25003120530432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚277 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚25819710949712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 NFDeviceStatsMonitor.o§üÕ¾n_train_batch_start/inactive_split.small_pool.peak‚482 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4751000 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3005187 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6961642 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚26165875289088 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚12090331 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 489558178304 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚26641364348928 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1745539 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚25492886331392 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6971554 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203421184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚25038815687680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 476167176704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4757675 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 _runtime‚ 261.250239565 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6971554 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5136010 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 490438061568 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3009477 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12097812 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12107564 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚25856583017472 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 476167176704 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚25856565978112 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 474648056364 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 trainer/global_step‚699 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 474561909642 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6961804 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5136008 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6961804 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4757427 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12097812 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25529075231744 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 % _timestamp‚1.7845021071868951e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12107564 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚26679303972864 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 476079355904 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚26203224616960 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1748198 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3009474 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚26203241656320 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚54 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚26203224616960 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚26203241656320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1747953 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 490259544064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5136010 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚26331127887754 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 _runtime‚ 261.250239565 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚6971554 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5136010 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 490438061568 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3009477 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚12097812 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12107564 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚25856583017472 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 476167176704 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚25856565978112 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 474648056364 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 trainer/global_step‚699 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 474561909642 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6961804 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5136008 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚6961804 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4757427 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚12097812 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25529075231744 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 % _timestamp‚1.7845021071868951e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12107564 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚26679303972864 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 476079355904 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚26203224616960 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1748198 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3009474 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚26203241656320 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚54 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚26203224616960 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚26203241656320 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1747953 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 490259544064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5136010 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚26331127887754 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚6971554 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203421184 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚25038815687680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 476167176704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47576756†É>¢0æ+ _wandbruntime‚261  _runtime‚261Þ8Füçö 1loss/FBt2/assignment_loss‚0.30594536751393253 /loss/FBt1/detection_loss‚21.555616296262347 _runtime‚ 261.250685326 _step‚55 2loss/SRqqt2/assignment_loss‚1.6201916245498074 1loss/FBt1/assignment_loss‚0.23282942661951114 /loss/FRt1/detection_loss‚0.4180872295849618 /loss/FRt2/detection_loss‚1.5334625733994256 1loss/SRqqt2/detection_loss‚1.2500638745284645 epoch‚0 trainer/global_step‚699 $ _timestamp‚1.784502107187885e+09 0loss/SRqqt1/detection_loss‚3.191432255842512 .loss/FBt2/detection_loss‚4.755725333573013 .loss/FRt2/assignment_loss‚3.72413370877328 1loss/SRqqt1/assignment_loss‚1.829517190133669 %loss/total_loss‚44.55265262257059 .loss/FRt1/assignment_loss‚4.13564774178967iXŸ˜èú 0loss/SRqqt1/detection_loss‚3.191432255842512 .loss/FBt2/detection_loss‚4.755725333573013 .loss/FRt2/assignment_loss‚3.72413370877328 1loss/SRqqt1/assignment_loss‚1.829517190133669 %loss/total_loss‚44.55265262257059 .loss/FRt1/assignment_loss‚4.13564774178967 1loss/FBt2/assignment_loss‚0.30594536751393253 /loss/FBt1/detection_loss‚21.555616296262347 _runtime‚ 261.250685326 _step‚55 2loss/SRqqt2/assignment_loss‚1.6201916245498074 1loss/FBt1/assignment_loss‚0.23282942661951114 /loss/FRt1/detection_loss‚0.4180872295849618 /loss/FRt2/detection_loss‚1.5334625733994256 1loss/SRqqt2/detection_loss‚1.2500638745284645 epoch‚0 trainer/global_step‚699 $ _timestamp‚1.784502107187885e+097 :¾}Gé‚(:¼ ã®õÒ¥óö¿ cpu‚26.073777116756904 network.recv‚118488 disk.md0.in‚0 disk.md0.out‚ 6.4765625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 40.42578125 network.sent‚673870% disk./.usageGB‚1167.9160346984863 disk.nvme1c1n1.in‚ 4.49609375 disk.nvme1c1n1.out‚ 76.484375 disk.nvme1n1.in‚ 4.49609375 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 40.42578125$ memory_percent‚98.75011444091797) disk./.usagePercent‚76.07334536384865 disk.nvme1n1.out‚ 76.484375 disk.dm-0.out‚441.7978515625# disk.nvme0n1.out‚382.0791015625! proc.memory.rssMB‚ 2797.234375) proc.memory.percent‚17.07296371459961 proc.cpu.threads‚17 disk.dm-0.in‚ 44.921875 disk.dm-1.out‚2.765625 disk.rbd0.out‚0% disk.nvme0c0n1.out‚382.0791015625% proc.memory.availableMB‚ 204.781258tƒœê‚(:‘ ã®õÒêÖÞÀ gpu.0.gpu‚68.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚188.932' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚62.97733333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,!þ30ë+ _wandbruntime‚279  _runtime‚279¹ úñÆìÀ ' lr-AdamW/pg2‚0.00010484785919558438 trainer/global_step‚749 % _timestamp‚1.7845021252222624e+09 _runtime‚ 279.287614615 _step‚56 ' lr-AdamW/pg1‚0.00010484785919558438cdÊíÄ ' lr-AdamW/pg1‚0.00010484785919558438 ' lr-AdamW/pg2‚0.00010484785919558438 trainer/global_step‚749 % _timestamp‚1.7845021252222624e+09 _runtime‚ 279.287614615 _step‚568œ$0î+ _wandbruntime‚279  _runtime‚279žèëKï O WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5494737 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚28034176611328 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚12943395 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28543620452352 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 509338988032 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5494739 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27663297298352 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28543515599360 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 509338988032 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7448658 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1869032 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚27663314337712 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 507715433606 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3219687 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26787878392832 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 509426801664 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5494739 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5088716 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28543515599360 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 509338988032 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7448658 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1869032 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚27663314337712 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 507715433606 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313357824 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3219687 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26787878392832 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 509426801664 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5494739 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5088716 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103185970 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 509338988032 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5494739 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27663297298352 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159788544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7458556 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28071542978560 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27348956439040 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 508477531772 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 508391385050 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚27700186405472 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5502058 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28071542978560 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 510017097728 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1871667 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27700169366112 trainer/global_step‚749 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28581664936448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26823598453760 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 525357985280 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28581664936448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176420352 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚28581560076288 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5502060 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26823573550080 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12970366 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1871424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5502060 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚58 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5095398 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7458556 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 510017097728 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7845021254733827e+09 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 510104918528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5095644 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3223974 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 525181564928ûºþ÷ó¯M I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5502058 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚12970366 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3223977 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚87820800 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚28071560017920 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5095398 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7458556 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 510017097728 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7845021254733827e+09 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 510104918528 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5095644 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3223974 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 104860160 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 525181564928 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚27348755115008 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313357824 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7458556 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28071542978560 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27348956439040 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 508477531772 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 508391385050 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚27700186405472 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5502058 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28071542978560 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 510017097728 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1871667 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27700169366112 trainer/global_step‚749 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159788544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A8DeviceStatsMonitor.on_train_batch_end/active.all.current‚9752 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28581664936448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26823598453760 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 525357985280 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 VCDeviceStatsMonitor!7ÜS¾ .on_train_batch_end/allocated_bytes.all.allocated‚28581664936448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176420352 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚28581560076288 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5502060 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚87820800 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26823573550080 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚12970366 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1871424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86146722 H?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚9750 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5502060 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313357824 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚58 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7845021435107753e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29902512012288 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 541631055624 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30445876555776 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5427522 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 _step‚61 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 559614682112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚30048548781336 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 559434060288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30445876555776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13815935 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205525504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3434184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚28572661158912 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29506900686352 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30445771702784 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5860789 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3434187 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1993586 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30445771702784 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29902512012288 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 543364543488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 543364543488 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7945410 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180621824 trainer/global_step‚799 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5860789 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 543276729856 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚29132070315520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1993338 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚9736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7845021435107753e+09 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚29902512012288 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 541631055624 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚87813632 G>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚9738 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30445876555776 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5427522 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 _step‚61 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094179591 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 104852992 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 559614682112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚30048548781336 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313357824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚11967 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 559434060288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30445876555776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚13815935 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205525504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚11967 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚11896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚9736 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚11896 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3434184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 104852992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚28572661158912 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 C:DeviceStatsMonitor.on_train_batch_start/active.all.current‚9738 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29506900686352 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30445771702784 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5860789 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3434187 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1993586 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30445771702784 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚29902512012288 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 543364543488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 543364543488 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚7945410 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180621824 trainer/global_step‚799 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5860789 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚87813632 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 543276729856 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚29132070315520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1993338 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29543789793472 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30483816179712 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3438474 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220198400 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29939878379520 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 544042660352 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5434258 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 542220860458 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7955308 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5868110 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 544042660352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29543772754112 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13833168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚29939861340160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5868108 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 543954839552 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚11967 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5434513 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1995784 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚29168676186624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 195294720 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3438477 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚7965058 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚29939861340160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5868110 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 543954839552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13833168 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5868108 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28608356316160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13823416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1996036 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30085993614570 $ _timestamp‚1.784502143758616e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7955308 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 _runtime‚ 297.821900358 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 543954839552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚11896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13833168 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚5868108 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28608356316160 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159788544 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13823416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1996036 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30085993614570 $ _timestamp‚1.784502143758616e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚11967 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚7955308 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚9750 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103186082 _runtime‚ 297.821900358 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚29543789793472 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30483816179712 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3438474 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 220198400 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚29939878379520 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 544042660352 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5434258 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094179591 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 542220860458 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7955308 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 104860160 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5868110 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 544042660352 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29543772754112 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚13833168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚29939861340160 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311680887>]Ã10‚+ _wandbruntime‚297  _runtime‚297BPÎxƒû _runtime‚ 297.822370719 _step‚63 /loss/FRt2/assignment_loss‚3.299195060196022 1loss/SRqqt2/detection_loss‚1.0683145453677898 %loss/total_loss‚37.74778180603295 2loss/SRqqt1/assignment_loss‚1.9528816773132844 1loss/SRqqt2/assignment_loss‚1.575023017958345 1loss/FBt1/assignment_loss‚0.07507763343935092 0loss/FRt1/detection_loss‚0.45412824899238935 1loss/SRqqt1/detection_loss‚3.2692901580369034 epoch‚0 % _timestamp‚1.7845021437595115e+09 /loss/FRt1/assignment_loss‚4.107424424844477 1loss/FBt2/assignment_loss‚0.16429267906138334 /loss/FRt2/detection_loss‚1.3162136392929824 .loss/FBt1/detection_loss‚16.60581622451162 trainer/global_step‚799 /loss/FBt2/detection_loss‚3.8601244970183926,3¬ „ÿ 2loss/SRqqt1/assignment_loss‚1.9528816773132844 1loss/SRqqt2/assignment_loss‚1.575023017958345 1loss/FBt1/assignment_loss‚0.07507763343935092 0loss/FRt1/detection_loss‚0.45412824899238935 1loss/SRqqt1/detection_loss‚3.2692901580369034 epoch‚0 % _timestamp‚1.7845021437595115e+09 /loss/FRt1/assignment_loss‚4.107424424844477 1loss/FBt2/assignment_loss‚0.16429267906138334 /loss/FRt2/detection_loss‚1.3162136392929824 .loss/FBt1/detection_loss‚16.60581622451162 trainer/global_step‚799 /loss/FBt2/detection_loss‚3.8601244970183926 _runtime‚ 297.822370719 _step‚63 /loss/FRt2/assignment_loss‚3.299195060196022 1loss/SRqqt2/detection_loss‚1.0683145453677898 %loss/total_loss‚37.74778180603295?ý\•^…‚(:Ó ¯õÒ̯ó¿ disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚396.76708984375 network.sent‚768956 disk.md0.out‚ 7.0078125 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 40.42578125 disk.nvme0n1.in‚ 40.42578125% disk.nvme1c1n1.out‚77.73681640625 disk.nvme1n1.in‚ 4.49609375# disk.nvme1n1.out‚77.73681640625 network.recv‚127297 disk.nvme1c1n1.in‚ 4.49609375# proc.memory.rssMB‚ 2835.01171875 disk.md0.in‚0$ disk.nvme0n1.out‚396.76708984375$ memory_percent‚98.98269176483154( proc.memory.availableMB‚ 166.67578125) proc.memory.percent‚17.30353832244873 cpu‚26.074694816919436 proc.cpu.threads‚17) disk./.usagePercent‚76.07333890352433% disk./.usageGB‚1167.9159355163574 disk.dm-0.in‚ 44.921875! disk.dm-0.out‚456.40771484375 disk.dm-1.out‚ 2.93212890625<ædž‚(:„ ¯õÒŠÅÀ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚152.772' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚50.924 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Òàp±H‡bC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPÑ nÜNˆbI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPQ~J‰bE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP|ëèÓJŠbE "Purity/2FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP w©RH‹bC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPM¼OxHŒbC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPu´µHbC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPƇ`HNŽbI &Purity/0FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP8 G†NbI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPn¸DXHbC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP„°;‘b6 jet/accuracy_1_of_1"trainer/global_step2:JP® Ý ;’b6 jet/accuracy_2_of_4"trainer/global_step2:JP¼úðJ“bE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPñ”ÒN”bI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPÉÁ¢‘H•bC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP‚ æH–bC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP r¤H—bC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP#&‡¹H˜bC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP•ª…rN™bI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP ÓRHšbC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP&åÛ‰J›bE "Purity/0FRt1SRqqt2FBt/event_purity"trainer/global_step2:JP•€²±HœbC Purity/*FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPÄQ¶NbI &Purity/0FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPŒ~ ºJžbE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPþ>ÂÄNŸbI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPšÏþJ bE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¸^LN¡bI &Purity/1FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPhkB“J¢bE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¶2Ú?J£bE "Purity/0FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPjvrN¤bI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP TI™H¥bC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP4චH¦bC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP¯â5H§bC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP£~_†N¨bI &Purity/2FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPÒÃzH©bC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP\*0JªbE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPKnôJ«bE "Purity/0FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPH»Z N¬bI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPô N­bI &Purity/1FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPV^î—N®bI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPÍh:@¯b; particle/accuracy_2_of_2"trainer/global_step2:JPá×J°bE "Purity/0FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¹¾³ÕH±bC Purity/*FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP¿˜Í~J²bE "Purity/2FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP€‹H³bC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPúÈÝ›H´bC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPíÿ`žJµbE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPh3J"J¶bE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP<áK¤H·bC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPM_,N¸bI &Purity/2FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP T-@¹b; particle/accuracy_1_of_4"trainer/global_step2:JP t‚JºbE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP+t?J»bE "Purity/2FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPÈ!M;¼b6 jet/accuracy_1_of_5"trainer/global_step2:JP™Ê`‚J½bE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP˜" QH¾bC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPÃG…J¿bE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPI°^HÀbC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP›&ˆHÁbC Purity/2FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP$ÿ°@Âb; particle/accuracy_4_of_5"trainer/global_step2:JP’H´ÞNÃbI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP'ÓžNÄbI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPòr²JÅbE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP´kžñJÆbE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP4ý!OJÇbE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPJNz^HÈbC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JP­ˆ>Æ@Éb; particle/accuracy_1_of_2"trainer/global_step2:JPv¿ÑJÊbE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÃôŽ£JËbE "Purity/*FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPƒµÝóHÌbC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPe f~HÍbC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPti³ JÜbE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP[ì‡ýJÝbE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPCípHÞbC Purity/2FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JP›Ø«ûJßbE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPJ$HàbC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPq@NábI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPjkÛ6HâbC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP¼D=JãbE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPxé½ÂJäbE "Purity/1FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPäLýNåbI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPÖ‘ kHæbC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPÜõTnJçbE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPŸéúôHèbC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPcåeNébI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP¿ xNêbI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP Þ”®HëbC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPÜ?Ú HìbC Purity/*FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPáÁ¡EJíbE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPR¯ØHîbC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPòM²NïbI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP…c©eNðbI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP)fþJñbE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPµù¬#HòbC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPÊ:(HóbC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPyø›ŸJôbE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP0ã˜`JõbE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPìniJöbE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP×U³J÷bE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPÁ²;®HøbC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP›ÒÁwHùbC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP2aÕHúbC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP*TiNûbI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPS^KJübE "Purity/0FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPl—92HýbC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPùgT;JþbE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPËϹÝHÿbC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPªŠ”üN€bI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JP8󌨰NñbI &Purity/1FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP KNòbI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP]ëñJóbE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPÓMÜÐNôbI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPË ³JõbE "Purity/0FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPJj¥HöbC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPqÑ]ÝN÷bI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPú0cJøbE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPÖ†‹hJùbE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP 2›uNúbI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP¨`„NûbI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JP¾9ÿ JübE "Purity/0FRt1SRqqt1FBt/event_purity"trainer/global_step2:JP·Æn–JýbE "Purity/1FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPè’ói;þb6 jet/accuracy_4_of_5"trainer/global_step2:JPøìÀ…JÿbE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPëoMšN€bI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPÞëz=HbC Purity/1FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPTÜö”J‚bE "Purity/2FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP[Ÿ w@ƒb; particle/accuracy_3_of_5"trainer/global_step2:JP¥5¤H„bC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPj8ˆN…bI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPiŒN†bI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP[ÉJ‡bE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPNkÂó@ˆb; particle/accuracy_1_of_5"trainer/global_step2:JP,ˆéKN‰bI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP wLjNŠbI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPš‹=Â;‹b6 jet/accuracy_2_of_3"trainer/global_step2:JPäð¼NŒbI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPýÙ}JbE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPšçHŽbC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JP_ ¸‹<b7 particle/sensitivity"trainer/global_step2:JPŠJ‘bE "Purity/*FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPP8GH’bC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPÀOÙJ“bE "Purity/1FRt0SRqqt0FBt/event_purity"trainer/global_step2:JPCÛJ”bE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPy­Î¸J•bE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPΈH–bC Purity/1FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JP².J—bE "Purity/2FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP¬ÿêé;˜b6 jet/accuracy_4_of_4"trainer/global_step2:JP*nØÿJ™bE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPc%±;šb6 jet/accuracy_3_of_3"trainer/global_step2:JP6é‘;›b6 jet/accuracy_2_of_2"trainer/global_step2:JPíä[¶;œb6 jet/accuracy_1_of_4"trainer/global_step2:JPÏ ÝØJbE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÿ7ÀNžbI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP¾ó@HŸbC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPÊâ¦/J bE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPïx6=H¡bC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP×þÀÅH¢bC Purity/0FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPÙ*BHN£bI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP)ô~N¤bI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPýiÞsN¥bI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPT9ß¾H¦bC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP +è¾0§+ _wandbruntime‚313  _runtime‚313Ê L=¨Žz ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1252838373184204 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3259868025779724 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4424569308757782 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06646525114774704 particle/accuracy_1_of_2‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6896630525588989 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.10929585993289948 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_2_of_5‚1 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0012531328247860074 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.03687652200460434 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0995393916964531 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3500848114490509 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 /particle/accuracy_3_of_4‚0.9918367266654968 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.15412087738513947 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9972007274627686 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5899999737739563 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.09166666865348816 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10427350550889969 , Purity/2FRt0SRqqt1FBt/FRt_purity‚0.09375 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9291666150093079 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9026389122009277 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.023394662886857986 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9907935857772827 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.021557779982686043 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.011619511991739273 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.127101868391037 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.21784353256225586 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4009819030761719 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.08998218923807144 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1785714328289032 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FRt_purity‚0 9 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.040033672004938126 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09880506247282028 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.12410327047109604 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7778255939483643 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962250590324402 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.001400560257025063 particle/accuracy_5_of_5‚1 epoch‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.35423168540000916 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9821978211402893 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.2185482680797577 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7647177577018738 particle/accuracy_2_of_3‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1297369748353958 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.22601531445980072 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.401274174451828 jet/accuracy_3_of_5‚0 *jet/accuracy_1_of_2‚0.5258470177650452 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.39087948203086853 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9976615905761719 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.08493858575820923 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.30767932534217834 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0007215007208287716 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.01544865407049656 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5899999737739563 (particle/accuracy‚0.7858420610427856 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.996092677116394 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08760470151901245 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.014009065926074982 ( Purity/1FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.13759338855743408 +"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.25 particle/accuracy_1_of_1‚1 jet/accuracy_2_of_5‚1 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.989871621131897 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09166666865348816 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ,jet/accuracy_3_of_4‚0.011688311584293842 particle/accuracy_1_of_3‚1 /particle/accuracy_3_of_3‚0.9979704022407532 'particle/f_score‚0.5389687418937683 _step‚64 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.34095197916030884 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.012565012089908123 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.12368807196617126 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9802631735801697 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3256043791770935 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.03699393942952156 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11684796959161758 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.17307692766189575 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3916214406490326 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11359649151563644 * Purity/0FRt1SRqqt2FBt/FBt_purity‚0.875 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.09224487841129303 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.022008363157510757 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.19404073059558868 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9924662709236145 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/1FRt0SRqqt*FBt/event_purity‚0.019216101616621017 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5655425190925598 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9604496955871582 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.34993135929107666 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.003932254854589701 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.8413349986076355 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.39267677068710327 7"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.45379838347435 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 trainer/global_step‚800 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9940665364265442 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.04050341621041298 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11886481195688248 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.2248869687318802 /particle/accuracy_4_of_4‚0.9563025236129761 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2003655731678009 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.40238863229751587 *jet/accuracy_1_of_3‚0.6073748469352722 jet/accuracy_5_of_5‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.31651294231414795 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚33.15194533854316 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9926573634147644 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.0677083358168602 _runtime‚ 313.213936777 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.40061163902282715 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9802249073982239 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.21780629456043243 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.03634907305240631 & Purity/2FRt*SRqqt2FBt/FRt_purity‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8"Purity/1FRt*SRqqt*FBt/event_purity‚0.015006716363132 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9888811707496643 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.01992689073085785 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.3780672252178192 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7403752207756042 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.19063027203083038 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_5‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.39340463280677795 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.143551304936409 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.20793350040912628 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9924153685569763 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.009418037720024586 particle/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.14166785776615143 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.000643500650767237 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.03689516335725784 +particle/sensitivity‚0.5829370021820068 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9839743375778198 # _timestamp‚1.78450215915032e+09 7"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.35478675365448 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.996092677116394 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.01609560288488865 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.45513367652893066 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_4‚0 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.20793350040912628 ,jet/accuracy_3_of_3‚0.007890579290688038 +jet/accuracy_2_of_2‚0.05307326838374138 *jet/accuracy_1_of_4‚0.5611419081687927 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.21616125106811523 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.01609560288488865 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.150438591837883 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9752976298332214 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.01572473905980587 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.08493529260158539 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.009535379707813263 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.14085212349891663 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9969373345375061 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.054663170129060745 +jet/accuracy_1_of_1‚0.15954385697841644 +jet/accuracy_2_of_4‚0.13746841251850128 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3496936857700348 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9964348673820496 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9882399439811707 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06646525114774704 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.13814036548137665 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3500848114490509 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.44677653908729553 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.277777761220932 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.34271860122680664 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.019437666982412338 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08324390649795532 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9955006837844849 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.44999998807907104 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.28787878155708313 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_2‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1785714328289032 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9787366986274719 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.29897648096084595 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.05702223256230354 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.19633257389068604 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0047533814795315266 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_1_of_4‚1 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.09204665571451187 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.029999999329447746 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1›Ân†%©’z +particle/sensitivity‚0.5829370021820068 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9839743375778198 # _timestamp‚1.78450215915032e+09 7"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.35478675365448 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.996092677116394 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.01609560288488865 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.45513367652893066 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_4‚0 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.20793350040912628 ,jet/accuracy_3_of_3‚0.007890579290688038 +jet/accuracy_2_of_2‚0.05307326838374138 *jet/accuracy_1_of_4‚0.5611419081687927 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.21616125106811523 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.01609560288488865 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.150438591837883 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9752976298332214 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.01572473905980587 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.08493529260158539 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.009535379707813263 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.14085212349891663 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9969373345375061 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.054663170129060745 +jet/accuracy_1_of_1‚0.15954385697841644 +jet/accuracy_2_of_4‚0.13746841251850128 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.3496936857700348 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9964348673820496 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 & Purity/1FRt1SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9882399439811707 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.06646525114774704 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.13814036548137665 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.3500848114490509 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.44677653908729553 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.277777761220932 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.34271860122680664 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.019437666982412338 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08324390649795532 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9955006837844849 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.44999998807907104 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.28787878155708313 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_2‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1785714328289032 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9787366986274719 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.29897648096084595 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.05702223256230354 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.19633257389068604 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0047533814795315266 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 particle/accuracy_1_of_4‚1 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.09204665571451187 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.029999999329447746 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 particle/accuracy_4_of_5‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.1252838373184204 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.3259868025779724 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4424569308757782 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.06646525114774704 particle/accuracy_1_of_2‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.6896630525588989 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.10929585993289948 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_2_of_5‚1 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0012531328247860074 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.03687652200460434 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.0995393916964531 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.3500848114490509 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 /particle/accuracy_3_of_4‚0.9918367266654968 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.15412087738513947 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9972007274627686 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5899999737739563 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.09166666865348816 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.10427350550889969 , Purity/2FRt0SRqqt1FBt/FRt_purity‚0.09375 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9291666150093079 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9026389122009277 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.023394662886857986 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9907935857772827 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.021557779982686043 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.011619511991739273 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.127101868391037 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.21784353256225586 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4009819030761719 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.08998218923807144 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1785714328289032 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FRt_purity‚0 9 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.040033672004938126 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.09880506247282028 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.12410327047109604 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.7778255939483643 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9962250590324402 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.001400560257025063 particle/accuracy_5_of_5‚1 epoch‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.35423168540000916 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9821978211402893 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.2185482680797577 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7647177577018738 particle/accuracy_2_of_3‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.1297369748353958 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.22601531445980072 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.401274174451828 jet/accuracy_3_of_5‚0 *jet/accuracy_1_of_2‚0.5258470177650452 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.39087948203086853 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9976615905761719 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.08493858575820923 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.30767932534217834 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0007215007208287716 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.01544865407049656 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5899999737739563 (particle/accuracy‚0.7858420610427856 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.996092677116394 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08760470151901245 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.014009065926074982 ( Purity/1FRt1SRqqt2FBt/FRt_purity‚0.5 @&Purity/0FRt1SRqqt2F÷aˆBt/event_proportion‚0.0004417782765813172 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.13759338855743408 +"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.25 particle/accuracy_1_of_1‚1 jet/accuracy_2_of_5‚1 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.989871621131897 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.09166666865348816 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ,jet/accuracy_3_of_4‚0.011688311584293842 particle/accuracy_1_of_3‚1 /particle/accuracy_3_of_3‚0.9979704022407532 'particle/f_score‚0.5389687418937683 _step‚64 & Purity/1FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.34095197916030884 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.012565012089908123 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.12368807196617126 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9802631735801697 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.3256043791770935 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.03699393942952156 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.11684796959161758 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.17307692766189575 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.3916214406490326 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11359649151563644 * Purity/0FRt1SRqqt2FBt/FBt_purity‚0.875 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.09224487841129303 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.022008363157510757 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.19404073059558868 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9924662709236145 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ;"Purity/1FRt0SRqqt*FBt/event_purity‚0.019216101616621017 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_4‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.5655425190925598 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9604496955871582 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.34993135929107666 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.003932254854589701 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 +particle/specificity‚0.8413349986076355 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.39267677068710327 7"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.45379838347435 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 trainer/global_step‚800 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9940665364265442 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.04050341621041298 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.11886481195688248 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.2248869687318802 /particle/accuracy_4_of_4‚0.9563025236129761 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2003655731678009 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.40238863229751587 *jet/accuracy_1_of_3‚0.6073748469352722 jet/accuracy_5_of_5‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.31651294231414795 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 )loss/val_total_loss‚33.15194533854316 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9926573634147644 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.0677083358168602 _runtime‚ 313.213936777 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.40061163902282715 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9802249073982239 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.21780629456043243 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.03634907305240631 & Purity/2FRt*SRqqt2FBt/FRt_purity‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8"Purity/1FRt*SRqqt*FBt/event_purity‚0.015006716363132 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9888811707496643 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.01992689073085785 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.3780672252178192 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7403752207756042 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.19063027203083038 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_5‚0 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.39340463280677795 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.143551304936409 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.20793350040912628 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9924153685569763 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.009418037720024586 particle/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.14166785776615143 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.000643500650767237 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.03689516335725784@™Ì·)Qª‚(:Æ ¯õÒòÔê¿ disk.dm-1.in‚0$ disk.nvme0n1.out‚416.05810546875% disk.nvme1c1n1.out‚78.55126953125 disk.nvme1n1.in‚ 4.49609375$ memory_percent‚99.20487403869629 network.recv‚131720 disk.dm-1.out‚ 3.09130859375! disk.nvme0c0n1.in‚ 40.42578125( proc.memory.percent‚17.4285888671875 cpu‚26.20197661875336 disk.md0.in‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚416.05810546875# disk.nvme1n1.out‚78.55126953125 proc.memory.rssMB‚2855.5 proc.cpu.threads‚17' disk./.usagePercent‚76.073352818069% disk./.usageGB‚1167.9161491394043 disk.dm-0.in‚ 44.921875 disk.rbd0.out‚0 disk.nvme0n1.in‚ 40.42578125 disk.nvme1c1n1.in‚ 4.49609375' proc.memory.availableMB‚ 130.2734375 network.sent‚814815 disk.md0.out‚7.515625! disk.dm-0.out‚475.26220703125®¿§›«‚(: ¯õÒ›–ÃÀ gpu.0.gpu‚64.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚158.296' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚52.76533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éZ ¿U¬‚(:Ê Ÿ¯õÒšúÑ¿ disk.nvme0n1.in‚ 40.42578125 disk.nvme1c1n1.in‚ 4.49609375$ memory_percent‚99.20508861541748( proc.memory.availableMB‚ 130.23828125# disk.nvme0n1.out‚419.7001953125 network.sent‚873709 network.recv‚136000) disk./.usagePercent‚76.06959067305687 disk.rbd0.in‚0% disk.nvme0c0n1.out‚419.7001953125& disk.nvme1c1n1.out‚207.14697265625 disk.md0.in‚0 disk.nvme1n1.in‚ 4.49609375 cpu‚26.189569016811994% disk./.usageGB‚1167.8583908081055 disk.dm-0.out‚ 479.1328125! disk.nvme0c0n1.in‚ 40.42578125$ disk.nvme1n1.out‚207.14697265625 disk.md0.out‚ 7.78515625 proc.memory.rssMB‚2855.5( proc.memory.percent‚17.4285888671875 proc.cpu.threads‚17 disk.dm-0.in‚ 44.921875 disk.dm-1.in‚0! disk.dm-1.out‚130.88427734375 disk.rbd0.out‚0{ý‹ò›­‚(: Ÿ¯õÒ‡Åÿ¿ gpu.0.gpu‚65.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚133.737' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚44.57899999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùH0®+ _wandbruntime‚331  _runtime‚331,ä£Yï½ _runtime‚ 331.70937207 _step‚65 & lr-AdamW/pg1‚0.0001121266025242727 & lr-AdamW/pg2‚0.0001121266025242727 trainer/global_step‚849 % _timestamp‚1.7845021776462543e+09'/ıÇ°Á & lr-AdamW/pg1‚0.0001121266025242727 & lr-AdamW/pg2‚0.0001121266025242727 trainer/global_step‚849 % _timestamp‚1.7845021776462543e+09 _runtime‚ 331.70937207 _step‚65AÐÌèQ0±+ _wandbruntime‚331  _runtime‚331Ãû'²¥O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 595768390823 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8609134 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚32743476929536 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 597634638336 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 615063777280 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15073439 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31282842285056 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 trainer/global_step‚849 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚33341111567872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8619275 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 614877071360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 597722659328 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 597722659328 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3742767 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚32743476929536 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2194296 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 _runtime‚ 331.710170141 _step‚66 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚32319010582064 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33341111567872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2194544 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚33341216628224 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚31897694452736 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3742764 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152µ,]¯'³©O LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8619275 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 614877071360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 597722659328 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 597722659328 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3742767 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚32743476929536 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2194296 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 _runtime‚ 331.710170141 _step‚66 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚32319010582064 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚33341111567872 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2194544 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚33341216628224 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚31897694452736 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3742764 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 595768390823 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚8609134 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚32743476929536 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 597634638336 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 615063777280 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15073439 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31282842285056 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 trainer/global_step‚849 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚33341111567872 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238B`ìF0´+ _wandbruntime‚331  _runtime‚331ÿ&ëÍ µ¯M S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33379261112320 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6471626 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 615584925184 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15100815 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6471628 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6471628 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5943825 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚33379261112320 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 598400776192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8619032 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 598312748032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8629187 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 615759041024 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚32355882649824 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2196771 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199019520 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 598312748032 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8629187 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31318512538624 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚32780860336128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚32355865610464 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 596530490845 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 _step‚67 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 596444342267 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15100815 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚33379156044800 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 598400776192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚849 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15090658 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15090658 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚32952309952731 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5944066 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31934097463808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174115840 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚ 331.961272809 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚32780843296768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840âœò¹&¶³M QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚8629187 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31318512538624 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚32780860336128 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚32355865610464 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 596530490845 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 _step‚67 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 596444342267 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15100815 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚33379156044800 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 598400776192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚849 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚15090658 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15090658 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚32952309952731 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5944066 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31934097463808 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174115840 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚ 331.961272809 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚32780843296768 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33379261112320 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6471626 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 615584925184 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15100815 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6471628 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6471628 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5943825 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚33379261112320 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 598400776192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8619032 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 598312748032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8629187 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 615759041024 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚32355882649824 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2196771 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199019520 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 598312748032CÏñý0·+ _wandbruntime‚331  _runtime‚331ª¦ÁǸú 2loss/SRqqt2/assignment_loss‚1.5221935640322715 0loss/FBt1/assignment_loss‚0.0820998106821715 %loss/total_loss‚34.00070816516654 _step‚68 /loss/FRt1/assignment_loss‚4.062636266398364 0loss/FRt2/assignment_loss‚3.2036413863384525 1loss/FBt2/assignment_loss‚0.22813168113275517 0loss/FRt1/detection_loss‚0.37102645633560377 /loss/FRt2/detection_loss‚1.3872137988784856 /loss/FBt1/detection_loss‚11.458188716226054 .loss/FBt2/detection_loss‚5.465862530793033 1loss/SRqqt1/detection_loss‚3.4241575103966895 epoch‚1 % _timestamp‚1.7845021778988814e+09 _runtime‚ 331.96182402 1loss/SRqqt1/assignment_loss‚1.838441939690174 1loss/SRqqt2/detection_loss‚0.9571145042624883 trainer/global_step‚849‰å¹þ /loss/FRt2/detection_loss‚1.3872137988784856 /loss/FBt1/detection_loss‚11.458188716226054 .loss/FBt2/detection_loss‚5.465862530793033 1loss/SRqqt1/detection_loss‚3.4241575103966895 epoch‚1 % _timestamp‚1.7845021778988814e+09 _runtime‚ 331.96182402 1loss/SRqqt1/assignment_loss‚1.838441939690174 1loss/SRqqt2/detection_loss‚0.9571145042624883 trainer/global_step‚849 2loss/SRqqt2/assignment_loss‚1.5221935640322715 0loss/FBt1/assignment_loss‚0.0820998106821715 %loss/total_loss‚34.00070816516654 _step‚68 /loss/FRt1/assignment_loss‚4.062636266398364 0loss/FRt2/assignment_loss‚3.2036413863384525 1loss/FBt2/assignment_loss‚0.22813168113275517 0loss/FRt1/detection_loss‚0.37102645633560377D²?…0Rº‚(:Ç ®¯õҥƆÀ disk.nvme1n1.in‚ 4.49609375 network.sent‚889912) disk./.usagePercent‚76.07784820988985$ disk./.usageGB‚1167.985164642334 disk.nvme0n1.in‚ 40.42578125 disk.dm-1.in‚0# memory_percent‚99.2053747177124 proc.memory.rssMB‚2855.5& disk.nvme0c0n1.out‚426.63134765625$ disk.nvme0n1.out‚426.63134765625 disk.nvme1c1n1.in‚ 4.49609375 disk.dm-0.in‚ 44.921875 disk.dm-1.out‚131.0634765625( proc.memory.availableMB‚ 130.19140625 cpu‚26.189245422415897! disk.nvme1n1.out‚ 207.77734375 network.recv‚139815 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 40.42578125# disk.nvme1c1n1.out‚ 207.77734375 disk.md0.in‚0! disk.dm-0.out‚485.81591796875 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.md0.out‚ 8.10546875( proc.memory.percent‚17.42858886718753D[o›»‚(: ®¯õҼĴÀ gpu.0.gpu‚60.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚201.151' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.05033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™Wi0¼+ _wandbruntime‚351  _runtime‚351àÑÈÀĽ¾ ' lr-AdamW/pg1‚0.00011212529149337631 ' lr-AdamW/pg2‚0.00011212529149337631 trainer/global_step‚899 $ _timestamp‚1.784502196993832e+09 _runtime‚ 351.05708049 _step‚699ÿ#Ⱦ ' lr-AdamW/pg2‚0.00011212529149337631 trainer/global_step‚899 $ _timestamp‚1.784502196993832e+09 _runtime‚ 351.05708049 _step‚69 ' lr-AdamW/pg1‚0.00011212529149337631E6œÜÒ0¿+ _wandbruntime‚351  _runtime‚351¤rýΫ'À¥O YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚33717479172096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3957264 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚33717692878848 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33067600147456 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6274405 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚34792194796935 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6830355 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 650092731392 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚34792297984761 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚34611795291136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚35243472731648 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 631572380160 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9105886 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 629597866231 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6830357 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚35243367671296 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚34162596930704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3957267 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚34611795291136 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚34162613970064 % _timestamp‚1.7845021969947016e+09 _step‚70 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 631572380160 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6274656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188803072 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15936241 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6830355 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _runtime‚ 351.057735751 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚35243472731648 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352üoíL Á©O VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213706752 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9116027 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2317141 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 631660401152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚34162613970064 % _timestamp‚1.7845021969947016e+09 _step‚70 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 631572380160 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6274656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ]ODeviceStatsMonitor.on_train_batch_star~‚ ct/inactive_split_bytes.small_pool.current‚ 188803072 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚15936241 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6830355 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _runtime‚ 351.057735751 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚35243472731648 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚33717479172096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3957264 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚33717692878848 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33067600147456 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6274405 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚34792194796935 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6830355 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 650092731392 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚34792297984761 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚34611795291136 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚35243472731648 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 631572380160 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9105886 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 629597866231 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6830357 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚35243367671296 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚34162596930704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3957267 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚34611795291136 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9125939 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 632338518016 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2319659 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 $ _timestamp‚1.784502197269869e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199019520 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15963617 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6837676 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3961554 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9115784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6281213 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174115840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚34649161658368 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 trainer/global_step‚899 _runtime‚ 351.333289093 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚35281517215744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚34829846004077 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 630273817675 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚35281517215744 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 632250489856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33103270401024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 632338518016 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6837678 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33103295304704 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15953460 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15963617 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9115784 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚34829742816139 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 650610459136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚35281412148224 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6837676 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904»¶Lb¸&IJM I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6837676 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚34649178697728 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 650784574976 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _step‚71 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6837678 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 632250489856 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9125939 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 632338518016 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2319659 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 $ _timestamp‚1.784502197269869e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199019520 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚15963617 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6837676 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3961554 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9115784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6281213 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174115840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚34649161658368 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 trainer/global_step‚899 _runtime‚ 351.333289093 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚35281517215744 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚34829846004077 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 630273817675 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚35281517215744 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 632250489856 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33103270401024 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 632338518016 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6837678 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚33103295304704 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚15953460 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚15963617 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9115784 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚34829742816139 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 650610459136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚35281412148224 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482G=‚ü0Å+ _wandbruntime‚351  _runtime‚351I1lÆû /loss/FBt2/detection_loss‚13.045416944385705 2loss/SRqqt2/assignment_loss‚1.4824395346159707 /loss/FBt1/detection_loss‚15.382449251142289 _step‚72 /loss/FRt2/detection_loss‚1.4726382546918197 $ _timestamp‚1.784502197270979e+09 _runtime‚ 351.333846734 &loss/total_loss‚45.551516962231304 /loss/FRt1/assignment_loss‚4.062370914630789 0loss/FRt2/assignment_loss‚3.1697041592156956 2loss/SRqqt1/assignment_loss‚1.8666860479074465 0loss/FBt1/assignment_loss‚0.1076597431737658 1loss/FBt2/assignment_loss‚0.08245961875319255 0loss/SRqqt1/detection_loss‚3.528352775844628 /loss/FRt1/detection_loss‚0.4295510396885066 1loss/SRqqt2/detection_loss‚0.9217886781814926 epoch‚1 trainer/global_step‚899Òp;.Çÿ &loss/total_loss‚45.551516962231304 /loss/FRt1/assignment_loss‚4.062370914630789 0loss/FRt2/assignment_loss‚3.1697041592156956 2loss/SRqqt1/assignment_loss‚1.8666860479074465 0loss/FBt1/assignment_loss‚0.1076597431737658 1loss/FBt2/assignment_loss‚0.08245961875319255 0loss/SRqqt1/detection_loss‚3.528352775844628 /loss/FRt1/detection_loss‚0.4295510396885066 1loss/SRqqt2/detection_loss‚0.9217886781814926 epoch‚1 trainer/global_step‚899 /loss/FBt2/detection_loss‚13.045416944385705 2loss/SRqqt2/assignment_loss‚1.4824395346159707 /loss/FBt1/detection_loss‚15.382449251142289 _step‚72 /loss/FRt2/detection_loss‚1.4726382546918197 $ _timestamp‚1.784502197270979e+09 _runtime‚ 351.333846734HÙ‘ï]TÈ‚(:É ½¯õÒúŽÅ¿ disk.dm-0.in‚ 44.921875& disk.nvme0c0n1.out‚429.58544921875$ disk.nvme0n1.out‚429.58544921875$ disk.nvme1c1n1.out‚ 208.564453125 disk.md0.in‚0$ memory_percent‚99.20754432678223 network.recv‚144331) disk./.usagePercent‚76.07788399014754 disk.md0.out‚ 8.4296875' proc.memory.availableMB‚ 129.8359375 proc.memory.rssMB‚2855.5( proc.memory.percent‚17.4285888671875 proc.cpu.threads‚17 disk.nvme1n1.in‚ 4.49609375" disk.nvme1n1.out‚ 208.564453125 cpu‚26.196435206787342! disk.dm-0.out‚488.73681640625 disk.dm-1.in‚0 disk.dm-1.out‚131.1689453125 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 40.42578125 disk.nvme0n1.in‚ 40.42578125 disk.nvme1c1n1.in‚ 4.49609375 network.sent‚939418% disk./.usageGB‚1167.9857139587402}vêªÉ‚(:… ½¯õÒõíñ¿ gpu.0.gpu‚46.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚111.738' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.246 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã÷0Ú0Ê+ _wandbruntime‚371  _runtime‚371A±–qÆËÀ _step‚73 ' lr-AdamW/pg1‚0.00011212308251922853 ' lr-AdamW/pg2‚0.00011212308251922853 trainer/global_step‚949 % _timestamp‚1.7845022178268695e+09 _runtime‚ 371.890043855íŠ)ÊÌÄ % _timestamp‚1.7845022178268695e+09 _runtime‚ 371.890043855 _step‚73 ' lr-AdamW/pg1‚0.00011212308251922853 ' lr-AdamW/pg2‚0.00011212308251922853 trainer/global_step‚949I™Á.0Í+ _wandbruntime‚371  _runtime‚3714ܞĕÎ¥O XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 trainer/global_step‚949 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36480113652736 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 665598142976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7196407 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚36006217358064 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚36006200318704 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37145623774720 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 665598142976 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37145728835072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6611918 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4171767 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7196407 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37145623774720 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 685035524608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9602638 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7196405 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9612779 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2440151 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6611685 ÔŽ.¯'Ï©O KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34852333106176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚35537393534464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7845022178279243e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7196407 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚36006217358064 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚36006200318704 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37145623774720 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 665598142976 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37145728835072 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚6611918 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4171767 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7196407 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37145623774720 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 685035524608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚9602638 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7196405 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚9612779 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2440151 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6611685 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 trainer/global_step‚949 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36480113652736 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 665598142976 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16826419 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚34888028263424 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36517480019968 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4176054 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36517480019968 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 664189441661 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205310976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 664103293083 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9612536 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35573785511936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7203728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9612536 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16816262 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6618650 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚34888053167104 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 685551937536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚949 _step‚75 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37183773319168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2442593 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16826419 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36517497059328 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 666188231680 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37183773319168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚36517497059328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 666188231680 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37183668251648 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6618393 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36043072386464 _runtime‚ 372.165829505 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2442339 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 685732344832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16816262 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9622691 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360ôOv;4Ò²M VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36517480019968 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4176054 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36517480019968 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 664189441661 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205310976 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 664103293083 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9612536 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚35573785511936 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7203728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9612536 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚16816262 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6618650 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚34888053167104 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 685551937536 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/se=ú„gment.small_pool.peak‚152 trainer/global_step‚949 _step‚75 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚37183773319168 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2442593 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚16826419 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36517497059328 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 666188231680 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37183773319168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚36517497059328 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 666188231680 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37183668251648 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6618393 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚36043072386464 _runtime‚ 372.165829505 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2442339 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 685732344832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚16816262 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9622691 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4176057 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 666276259840 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7203726 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚36707278867485 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16826419 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚34888028263424 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157K*ÕU0Ó+ _wandbruntime‚372  _runtime‚372±XÔý 1loss/SRqqt1/detection_loss‚2.4129128799405892 /loss/FRt2/assignment_loss‚3.012032606355522 % _timestamp‚1.7845022181035454e+09 _runtime‚ 372.166354216 0loss/FRt1/assignment_loss‚3.9722413951759443 2loss/SRqqt2/assignment_loss‚1.4879073440699893 0loss/FRt1/detection_loss‚0.44359319922899665 epoch‚1 2loss/SRqqt1/assignment_loss‚1.7099035231603517 1loss/SRqqt2/detection_loss‚0.9762172802398575 /loss/FBt1/detection_loss‚13.330014207398476 /loss/FBt2/detection_loss‚3.6654160390017014 %loss/total_loss‚32.95535730785832 trainer/global_step‚949 _step‚76 1loss/FBt1/assignment_loss‚0.39666391666024636 1loss/FBt2/assignment_loss‚0.19025986441208617 .loss/FRt2/detection_loss‚1.358195052214559F šxÕ /loss/FBt1/detection_loss‚13.330014207398476 /loss/FBt2/detection_loss‚3.6654160390017014 %loss/total_loss‚32.95535730785832 trainer/global_step‚949 _step‚76 1loss/FBt1/assignment_loss‚0.39666391666024636 1loss/FBt2/assignment_loss‚0.19025986441208617 .loss/FRt2/detection_loss‚1.358195052214559 1loss/SRqqt1/detection_loss‚2.4129128799405892 /loss/FRt2/assignment_loss‚3.012032606355522 % _timestamp‚1.7845022181035454e+09 _runtime‚ 372.166354216 0loss/FRt1/assignment_loss‚3.9722413951759443 2loss/SRqqt2/assignment_loss‚1.4879073440699893 0loss/FRt1/detection_loss‚0.44359319922899665 epoch‚1 2loss/SRqqt1/assignment_loss‚1.7099035231603517 1loss/SRqqt2/detection_loss‚0.9762172802398575LòyµâRÖ‚(:Ç ̯õ҇ݼ¿ network.sent‚ 1.002795e+06 disk.dm-1.in‚0! disk.nvme0c0n1.in‚ 40.42578125$ disk.nvme1c1n1.out‚ 209.498046875# memory_percent‚99.2079496383667 disk.dm-0.in‚ 44.921875 disk.rbd0.out‚0$ disk.nvme0c0n1.out‚ 431.646484375 disk.nvme0n1.in‚ 40.42578125 disk.md0.in‚0 disk.md0.out‚ 8.9609375( proc.memory.availableMB‚ 129.76953125( proc.memory.percent‚17.4285888671875( disk./.usagePercent‚76.0780054939393% disk./.usageGB‚1167.9875793457031 disk.dm-0.out‚ 490.490234375 disk.dm-1.out‚ 131.310546875 disk.rbd0.in‚0 cpu‚26.225498147336605 proc.cpu.threads‚17 network.recv‚148930" disk.nvme0n1.out‚ 431.646484375 disk.nvme1c1n1.in‚ 4.49609375 disk.nvme1n1.in‚ 4.49609375" disk.nvme1n1.out‚ 209.498046875 proc.memory.rssMB‚2855.5Ðmèì›×‚(: ̯õÒÙç›À gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚122.272' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.757333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*í¥UØ‚(:Ê Û¯õÒóáÏ¿$ disk.nvme0c0n1.out‚ 457.626953125% disk.nvme1c1n1.out‚210.2998046875 disk.nvme1n1.in‚ 4.49609375 disk.md0.out‚ 9.27734375 disk.dm-1.in‚0 disk.dm-1.out‚ 131.408203125 network.recv‚153482 disk.rbd0.in‚0# disk.nvme1n1.out‚210.2998046875 disk.md0.in‚0 disk.dm-0.in‚ 44.921875 disk.dm-0.out‚516.4775390625& proc.memory.availableMB‚ 4.46484375 cpu‚26.26351082202455 disk.rbd0.out‚0 disk.nvme0n1.in‚ 40.42578125" disk.nvme0n1.out‚ 457.626953125 disk.nvme1c1n1.in‚ 4.49609375 proc.memory.rssMB‚2855.5 proc.cpu.threads‚17) disk./.usagePercent‚76.07800176682913% disk./.usageGB‚1167.9875221252441! disk.nvme0c0n1.in‚ 40.42578125$ memory_percent‚99.97274875640869( proc.memory.percent‚17.4285888671875 network.sent‚ 1.066074e+06Z›Ù‚(: Û¯õÒ¨¬ü¿ gpu.0.gpu‚48.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚200.533' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.84433333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ï1F0Ú+ _wandbruntime‚392  _runtime‚392›ï55ÆÛÀ _runtime‚ 392.442916947 _step‚77 ' lr-AdamW/pg1‚0.00011211997563721051 ' lr-AdamW/pg2‚0.00011211997563721051 trainer/global_step‚999 % _timestamp‚1.7845022383797617e+09›óê©ÊÜÄ ' lr-AdamW/pg1‚0.00011211997563721051 ' lr-AdamW/pg2‚0.00011211997563721051 trainer/global_step‚999 % _timestamp‚1.7845022383797617e+09 _runtime‚ 392.442916947 _step‚77MvÙ/ƒ0Ý+ _wandbruntime‚392  _runtime‚392«Õ}¹¯'Þ©O GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4386264 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 719974766592 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7562455 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚37849803706704 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4386267 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17671988 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10109531 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39047879878144 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚39047984938496 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 699535884800 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7562457 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7562457 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚37356881126400 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚39047879878144 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7562455 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚38348432014336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6949213 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2563194 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184608768 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 % _timestamp‚1.7845022383807495e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10099390 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚38348449053696 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚999 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚38547060523751 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚37849820746064 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 719790157824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚38547163711577 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 _step‚78 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565184 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚38348432014336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚38348449053696 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚36637115872256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 699447863808 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚39047984938496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10109531 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 697256817047 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17661845 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚37357090638848jö´¥³'ß­O NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚38547060523751 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚37849820746064 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 719790157824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚38547163711577 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 _step‚78 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565184 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚38348432014336 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚38348449053696 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚36637115872256 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 699447863808 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚39047984938496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10109531 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 697256817047 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17661845 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚37357090638848 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4386264 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 719974766592 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚7562455 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚37849803706704 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4386267 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17671988 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10109531 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39047879878144 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚39047984938496 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 699535884800 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7562457 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7562457 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚37356881126400 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚39047879878144 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7562455 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚38348432014336 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚6949213 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2563194 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184608768 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 % _timestamp‚1.7845022383807495e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10099390 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚38348449053696 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚999 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743NSU‡ 0à+ _wandbruntime‚392  _runtime‚392ÒÙKßsá³M G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17679064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6956172 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10109288 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10109288 % _timestamp‚1.7845022386501222e+09 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6955930 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 700125973504 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199019520 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 _step‚79 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 TEDeviceStatsMonitor.on_train_batch_end/allocateYOSsFd_bytes.large_pool.peak‚ 5878632448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174115840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4390557 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17689221 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚37393477220864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚37393278201344 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _runtime‚ 392.713491903 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4390554 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚36672786125824 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 720492075520 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 trainer/global_step‚999 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚38584711730893 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚36672811029504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚39086029422592 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7569776 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚39086029422592³"W½&â·M C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17679064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7569776 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚39086029422592 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚17679064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6956172 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10109288 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10109288 % _timestamp‚1.7845022386501222e+09 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6955930 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 700125973504 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199019520 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 _step‚79 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174115840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4390557 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17689221 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚37393477220864 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚37393278201344 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _runtime‚ 392.713491903 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4390554 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚36672786125824 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 720492075520 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 trainer/global_step‚999 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚38584711730893 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚36672811029504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚39086029422592 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚40950241041920 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 _runtime‚ 412.958157928 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚40950135981568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7845022588948944e+09 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 733473626624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚38421873734656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7928507 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚39176629297152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10606283 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚40216767415296 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 733385605632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚39693424134064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚40216750375936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚40216750375936 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 733385605632 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 733473626624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40950135981568 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10596142 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚40424596574985 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2686393 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2686177 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚40424493387159 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 _step‚82 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4600767 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4600764 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18534790 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182511616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 731086292455 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7928505 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10606283ÚãJnçí®O XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚40950241041920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚38421848830976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10596142 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7287160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1049 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚40950241041920 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 _runtime‚ 412.958157928 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚ÉUDÍ!482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚40950135981568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7845022588948944e+09 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 733473626624 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚38421873734656 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7928507 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚39176629297152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚10606283 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚40216767415296 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 733385605632 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚39693424134064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚40216750375936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚40216750375936 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 733385605632 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 733473626624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 195939840 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚40950135981568 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10596142 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚40424596574985 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2686393 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2686177 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚40424493387159 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 _step‚82 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4600767 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4600764 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚18534790 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182511616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 731086292455 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7928505 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10606283 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 754780466176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10606040 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚40988180458496 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4605057 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10616195 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18541866 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211602432 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚40462041406363 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7935826 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7293555 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 734151743488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 731848392477 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚40254116743168 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚40254133782528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2688741 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18552023 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39213017734656 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10616195 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 731762243899 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚40988285526016 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18552023 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39213229337088 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚39730296201824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7935828 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚40254116743168 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2688501 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0Ú0cñ½&ð·M J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 _step‚83 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10616195 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚18541866 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211602432 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚40462041406363 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7935826 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7293555 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 734151743488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 731848392477 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚40254116743168 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚40254133782528 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2688741 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚18552023 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39213017734656 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10616195 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 731762243899 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚40988285526016 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚18552023 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 195939840 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39213229337088 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚39730296201824 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7935828 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚40254116743168 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2688501 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4605054 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚38457568891904 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40462144594301 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _runtime‚ 413.228738375 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚40254133782528 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚40988285526016 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 734151743488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10606040 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚40988180458496 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4605057 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11103035 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40206631597056 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19397592 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42852392084992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚42085085776896 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 767323347456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205318144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42085085776896 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7625007 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 197829632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40206606693376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8294557 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11103035 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _step‚86 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 764915767863 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 789932066304 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚42302029438393 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 767411368448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41537027522064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8294555 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚42852392084992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42085068737536 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚40996358345216üS×*´'û®O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 764915767863 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 789932066304 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚42302029438393 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 767411368448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41537027522064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8294555 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚42852392084992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42085068737536 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚40996358345216 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42085068737536 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8294555 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4815264 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180414464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 765001916329 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚42301926250567 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚40996563663360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 _runtime‚ 433.142642559 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7625265 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11092894 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚42852497145344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4815267 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2809998 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 789751651840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 767411368448 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2809743 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41537010482704 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11103035 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚40206631597056 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19397592 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42852392084992 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚42085085776896 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 767323347456 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205318144 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42085085776896 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7625007 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 197829632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40206606693376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8294557 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11103035 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _step‚86VžèªÉ0ü+ _wandbruntime‚433  _runtime‚433°è¸vº&ý´M QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8301876 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7631851 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 768001457152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2812546 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8301878 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4819554 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42122435104768 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 768089485312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40242301850624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40242326754304 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42890541629440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42890436561920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203213824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19414825 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41573882550464 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19404668 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 790631000064 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41573899589824 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2812297 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178310144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11112947 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11102792 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41032957754368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7632103 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4819557 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 trainer/global_step‚1099 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19414825 % _timestamp‚1.7845022793452897e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8301876 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 768001457152 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42890541629440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42122435104768 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _runtime‚ 433.408640951»ÊÀ¹ þ¸M I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7631851 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 768001457152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2812546 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚8301878 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 JBDeviceStatsMonitor.on_train_batch_eµ—­nd/segment.small_pool.allocated‚152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚4819554 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚42122435104768 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 768089485312 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40242301850624 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40242326754304 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42890541629440 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42890436561920 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203213824 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19414825 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚41573882550464 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19404668 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 790631000064 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41573899589824 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2812297 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178310144 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11112947 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11102792 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚41032957754368 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7632103 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4819557 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 trainer/global_step‚1099 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19414825 % _timestamp‚1.7845022793452897e+09 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8301876 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 768001457152 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42890541629440 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42122435104768 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _runtime‚ 433.408640951 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8301876 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238W1õ!r0ÿ+ _wandbruntime‚433  _runtime‚433Lì´€ü 0loss/FRt2/assignment_loss‚2.6462466073937194 /loss/FRt2/detection_loss‚1.5127086580993585 % _timestamp‚1.7845022793463306e+09 _runtime‚ 433.409229911 0loss/FBt1/assignment_loss‚0.2137728004954814 1loss/SRqqt1/detection_loss‚3.0780465690525687 1loss/SRqqt2/detection_loss‚0.8286652216810872 %loss/total_loss‚27.46352258736649 trainer/global_step‚1099 /loss/FRt1/assignment_loss‚3.969755996869305 2loss/SRqqt2/assignment_loss‚1.3757003617932528 1loss/FBt2/assignment_loss‚0.07886573047662984 /loss/FRt1/detection_loss‚0.5076097903555941 .loss/FBt1/detection_loss‚7.957319598776503 epoch‚1 _step‚88 2loss/SRqqt1/assignment_loss‚1.7576673276591719 /loss/FBt2/detection_loss‚3.5371639247138225c˼P€ .loss/FBt1/detection_loss‚7.957319598776503 epoch‚1 _step‚88 2loss/SRqqt1/assignment_loss‚1.7576673276591719 /loss/FBt2/detection_loss‚3.5371639247138225 0loss/FRt2/assignment_loss‚2.6462466073937194 /loss/FRt2/detection_loss‚1.5127086580993585 % _timestamp‚1.7845022793463306e+09 _runtime‚ 433.409229911 0loss/FBt1/assignment_loss‚0.2137728004954814 1loss/SRqqt1/detection_loss‚3.0780465690525687 1loss/SRqqt2/detection_loss‚0.8286652216810872 %loss/total_loss‚27.46352258736649 trainer/global_step‚1099 /loss/FRt1/assignment_loss‚3.969755996869305 2loss/SRqqt2/assignment_loss‚1.3757003617932528 1loss/FBt2/assignment_loss‚0.07886573047662984 /loss/FRt1/detection_loss‚0.5076097903555941Xþí™»X‚‚(:Í ˆ°õÒìì¿( proc.memory.percent‚17.4285888671875 cpu‚26.360111534889498 proc.cpu.threads‚17 network.sent‚ 1.194134e+06$ disk./.usageGB‚1167.987506866455 disk.nvme0n1.in‚ 45.30859375& disk.nvme1c1n1.out‚212.24853515625 disk.md0.in‚0 network.recv‚166158) disk./.usagePercent‚76.07800077293308 disk.dm-0.out‚531.5009765625% disk.nvme0c0n1.out‚473.3896484375 disk.nvme1n1.in‚ 5.98828125 proc.memory.rssMB‚2855.5 disk.dm-0.in‚ 51.296875 disk.dm-1.in‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 45.30859375# disk.nvme0n1.out‚473.3896484375 disk.nvme1c1n1.in‚ 5.98828125$ disk.nvme1n1.out‚212.24853515625$ memory_percent‚99.97358322143555! disk.dm-1.out‚131.74267578125 disk.rbd0.out‚0 disk.md0.out‚ 10.359375$ proc.memory.availableMB‚4.328125$,cƒ‚(:… ˆ°õÒ‘ÿÅÀ gpu.0.gpu‚50.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚164.934' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚54.978 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»9`çX„‚(:Í —°õÒÒþá¿ disk.rbd0.in‚0$ memory_percent‚99.97267723083496 proc.memory.rssMB‚2855.5( proc.memory.percent‚17.4285888671875) disk./.usagePercent‚76.07799456108279 disk.dm-0.in‚ 51.296875 disk.dm-0.out‚ 552.830078125 disk.rbd0.out‚0 cpu‚26.38942458918594% disk./.usageGB‚1167.9874114990234 disk.nvme1c1n1.in‚ 5.98828125& disk.nvme1c1n1.out‚212.93408203125$ disk.nvme0c0n1.out‚ 494.771484375% proc.memory.availableMB‚ 4.4765625 proc.cpu.threads‚17 network.recv‚170709" disk.nvme0n1.out‚ 494.771484375 disk.nvme1n1.in‚ 5.98828125$ disk.nvme1n1.out‚212.93408203125 disk.md0.in‚0 disk.md0.out‚ 10.66015625! disk.dm-1.out‚131.84423828125! disk.nvme0c0n1.in‚ 45.30859375 network.sent‚ 1.256701e+06 disk.nvme0n1.in‚ 45.30859375 disk.dm-1.in‚0Ýùœ…‚(:‘ —°õÒ²ÛžÀ gpu.0.gpu‚80.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚202.874' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.62466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UW:¬0†+ _wandbruntime‚453  _runtime‚453ÕheƇÀ trainer/global_step‚1149 % _timestamp‚1.7845022995566924e+09 _runtime‚ 453.61982235 _step‚89 ' lr-AdamW/pg1‚0.00011210526811346952 ' lr-AdamW/pg2‚0.00011210526811346952Á~d6ÊˆÄ trainer/global_step‚1149 % _timestamp‚1.7845022995566924e+09 _runtime‚ 453.61982235 _step‚89 ' lr-AdamW/pg1‚0.00011210526811346952 ' lr-AdamW/pg2‚0.00011210526811346952Y…6”0‰+ _wandbruntime‚453  _runtime‚453K˜ƒ\¯'Š©O KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚42816102972416 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44754648188416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5029767 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚1149 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44179462301801 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚43953387099136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43380630910064 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 798745243271 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚43953404138496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 801261089280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11589646 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215803904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _runtime‚ 453.620761301 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 801261089280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚90 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚44754753248768 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8660605 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11599787 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 190900224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 $ _timestamp‚1.784502299557637e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8660607 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7963458 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2933694 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 824929316864 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚44754648188416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2933934 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41991364555776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 197829632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚44754753248768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20260394 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 824738416640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 801349110272 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5029764 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 798831391737 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44179359113975i™ÓÀ³'‹­O [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚43953387099136 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 824738416640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 801349110272 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5029764 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 798831391737 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚44179359113975 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚42816102972416 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44754648188416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5029767 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚1149 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44179462301801 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚43953387099136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43380630910064 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 798745243271 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚43953404138496 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 801261089280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11589646 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215803904 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _runtime‚ 453.620761301 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 801261089280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚90 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚44754753248768 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8660605 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11599787 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 190900224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 $ _timestamp‚1.784502299557637e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚8660607 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7963458 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚2933694 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 824929316864 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚44754648188416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2933934 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41991364555776 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 197829632 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚44754753248768 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚20260394 IDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 801939198976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚43990770505728 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 R?DeviceStatsMonitor.on_train_batch_end/allocn§išÅ"ated_bytes.all.freed‚44792692665344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 197829632 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20277627 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11609699 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 trainer/global_step‚1149 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20277627 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7970429 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44217010321117 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207408128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7970177 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5034054 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚42027084616704 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 825438512640 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 799507343293 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚44792797732864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚43990770505728 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11599544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚43417502977824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚43990753466368 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8667926 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 825621017088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8667926 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8667926 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 801939198976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚43990770505728 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚44792692665344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 197829632 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚20277627 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11609699 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 trainer/global_step‚1149 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚20277627 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7970429 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚44217010321117 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207408128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7970177 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5034054 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚42027084616704 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 825438512640 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 799507343293 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚44792797732864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚43990770505728 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11599544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚43417502977824 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482[š i80+ _wandbruntime‚453  _runtime‚453E”ú 0loss/FRt2/assignment_loss‚2.6572372034444647 /loss/FBt1/detection_loss‚6.0315996554009965 .loss/FBt2/detection_loss‚6.940562950536197 trainer/global_step‚1149 0loss/FRt1/assignment_loss‚3.9985023029451447 .loss/FBt2/assignment_loss‚0.24042771227505 /loss/FRt2/detection_loss‚1.3638292456989236 &loss/total_loss‚28.880606934381404 epoch‚1 $ _timestamp‚1.784502299826588e+09 _runtime‚ 453.89258556 _step‚92 2loss/SRqqt1/assignment_loss‚1.7838040297623161 0loss/FRt1/detection_loss‚0.49409682673703736 1loss/SRqqt1/detection_loss‚2.7283147298321966 1loss/SRqqt2/detection_loss‚0.8853590360801238 2loss/SRqqt2/assignment_loss‚1.4262820825498137 0loss/FBt1/assignment_loss‚0.3305911591191389&z›‘þ 2loss/SRqqt1/assignment_loss‚1.7838040297623161 0loss/FRt1/detection_loss‚0.49409682673703736 1loss/SRqqt1/detection_loss‚2.7283147298321966 1loss/SRqqt2/detection_loss‚0.8853590360801238 2loss/SRqqt2/assignment_loss‚1.4262820825498137 0loss/FBt1/assignment_loss‚0.3305911591191389 0loss/FRt2/assignment_loss‚2.6572372034444647 /loss/FBt1/detection_loss‚6.0315996554009965 .loss/FBt2/detection_loss‚6.940562950536197 trainer/global_step‚1149 0loss/FRt1/assignment_loss‚3.9985023029451447 .loss/FBt2/assignment_loss‚0.24042771227505 /loss/FRt2/detection_loss‚1.3638292456989236 &loss/total_loss‚28.880606934381404 epoch‚1 $ _timestamp‚1.784502299826588e+09 _runtime‚ 453.89258556 _step‚92\ltòX’‚(:Í ¦°õÒ¬úø¿" disk.nvme0n1.out‚ 500.361328125 disk.nvme1n1.in‚ 5.98828125& proc.memory.availableMB‚ 4.41015625 proc.memory.rssMB‚2855.5 network.sent‚ 1.261881e+06 disk.nvme1c1n1.in‚ 5.98828125$ disk.nvme1n1.out‚213.68701171875 disk.md0.in‚0 disk.md0.out‚ 11.171875 disk.dm-1.in‚0$ memory_percent‚99.97308254241943( proc.memory.percent‚17.4285888671875) disk./.usagePercent‚76.06988039375459! disk.dm-0.out‚557.93798828125 disk.dm-1.out‚ 131.982421875 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 500.361328125 disk.dm-0.in‚ 51.296875% disk./.usageGB‚1167.8628387451172 disk.nvme0n1.in‚ 45.30859375& disk.nvme1c1n1.out‚213.68701171875 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 45.30859375 cpu‚26.414784136155692 proc.cpu.threads‚17 network.recv‚174702'Ðœ“‚(:‘ ¦°õÒ«ÔÀ gpu.0.gpu‚90.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚112.256' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.41866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„ü§0”+ _wandbruntime‚473  _runtime‚473¾øK.Ç•Á % _timestamp‚1.7845023198958461e+09 _runtime‚ 473.959006879 _step‚93 ' lr-AdamW/pg1‚0.00011209857024140573 ' lr-AdamW/pg2‚0.00011209857024140573 trainer/global_step‚1199@ Ë–Å ' lr-AdamW/pg2‚0.00011209857024140573 trainer/global_step‚1199 % _timestamp‚1.7845023198958461e+09 _runtime‚ 473.959006879 _step‚93 ' lr-AdamW/pg1‚0.00011209857024140573]+áž0—+ _wandbruntime‚473  _runtime‚4731â°°'˜ªO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 859846379008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚94 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚46657009352192 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚44635993700864 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚46056791977383 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45224234298064 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8301738 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 832660867145 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 197829632 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚46657009352192 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 835286852096 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚44635782091264 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46656904291840 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 832574718679 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9026657 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12096539 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3057474 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21113053 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12086398 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9026655 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 859659673088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143=;qX™®O C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 859846379008 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚94 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚46657009352192 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚44635993700864 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚46056791977383 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45224234298064 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8301738 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 832660867145 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 197829632 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚46657009352192 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 835286852096 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚44635782091264 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚46656904291840 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 832574718679 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚9026657 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12096539 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3057474 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21113053 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12086398 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9026655 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 859659673088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466^Ùˆ"0š+ _wandbruntime‚474  _runtime‚474ïú"»º&›´M TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 835876940800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21140429 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21140429 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21130272 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8308653 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9033978 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12106451 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9033976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 860545665024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚46695053836288 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 860358966272 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚44672176541696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43811817575424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186698752 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12096296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12096296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12106451 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 835876940800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8308402 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚44672388144128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45859088867328 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5248557 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 Hƾ&œ¸M ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚45261106365824 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 833250670123 trainer/global_step‚1199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚45859071827968 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 % _timestamp‚1.7845023201602802e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 835964968960 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9033976 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚43811842479104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 835964968960 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5248554 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚45859088867328 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚44672176541696 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚43811817575424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186698752 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12096296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12096296 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12106451 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 835876940800 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8308402 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚44672388144128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚45859088867328 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5248557 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 835876940800 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21140429 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21140429 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21130272 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8308653 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9033978 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚12106451 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9033976 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 860545665024 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚46695053836288 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 860358966272 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 869136572928 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 866404194087 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚48559160395264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚47934224840791 _step‚98 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚47067820646704 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565184 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8639059 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21985998 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5458767 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3180292 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5458764 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 869136572928 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8638802 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 197829632 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45560880280576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207415296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12583150 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 _runtime‚ 494.118658634 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48559160395264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46455487791104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚48559265455616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12593291 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 894607510528 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 866490342553 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 894790022144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182511616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9392705 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12593291 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚47690023822336 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 869224593920 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9392705 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45560905184256 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚47690023822336 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚46455695206400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21975855 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚47067837686064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 % _timestamp‚1.7845023400546215e+09 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47934328028617 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808Çâ ´'§®O YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 866404194087 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚48559160395264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚47934224840791 _step‚98 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚47067820646704 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565184 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8639059 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚21985998 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5458767 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3180292 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5458764 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 869136572928 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8638802 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 197829632 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚45560880280576 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207415296 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12583150 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 _runtime‚ 494.118658634 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48559160395264 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46455487791104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚48559265455616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚12593291 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 894607510528 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 866490342553 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 894790022144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182511616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9392705 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12593291 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚47690023822336 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 869224593920 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9392705 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚45560905184256 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚47690023822336 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚46455695206400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21975855 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚47067837686064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 % _timestamp‚1.7845023400546215e+09 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47934328028617 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 869224593920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚1249 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3180038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚12583150 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚47690040861696 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 869136572928bÅå0¨+ _wandbruntime‚494  _runtime‚494W»€cº&©´M QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5463054 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3182666 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209505280 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12603203 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47727390189568 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚48597309939712 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48597204872192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5463057 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚47104709753824 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚47104692714464 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 867166294109 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46492085787136 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚47727407228928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12593048 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22003231 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9400026 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8645488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 895300844032 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9400026 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47971772859995 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1249 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 869814682624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45596600341504 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22003231 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12593048 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45596575437824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235DÕè!ª¸M ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22003231 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12593048 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚45596575437824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚235 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5463054 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3182666 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209505280 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚12603203 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.fmÝÍþreed‚47727390189568 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚48597309939712 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48597204872192 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5463057 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚47104709753824 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚47104692714464 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 867166294109 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚46492085787136 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚47727407228928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚12593048 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22003231 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9400026 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8645488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 895300844032 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9400026 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47971772859995 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚1249 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 869814682624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚45596600341504 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0cjn¯0«+ _wandbruntime‚494  _runtime‚494Ù8UI¬ÿ &loss/total_loss‚27.749231608632034 % _timestamp‚1.7845023403216054e+09 _step‚100 2loss/SRqqt2/assignment_loss‚1.2727929431984195 1loss/SRqqt2/detection_loss‚0.8168500099350084 epoch‚1 trainer/global_step‚1249 _runtime‚ 494.384477304 /loss/FRt1/assignment_loss‚3.984745115569875 1loss/FBt2/assignment_loss‚0.16053705502019966 /loss/FBt1/detection_loss‚10.586306899916508 0loss/FRt2/assignment_loss‚2.6577999071578757 2loss/SRqqt1/assignment_loss‚1.7896771533799565 1loss/FBt1/assignment_loss‚0.21335537488132883 /loss/FRt1/detection_loss‚0.5263496787299083 /loss/FRt2/detection_loss‚1.3086085120991384 0loss/SRqqt1/detection_loss‚2.391826265219422 /loss/FBt2/detection_loss‚2.0403826935243927ŠÅØ. ­ƒ 2loss/SRqqt2/assignment_loss‚1.2727929431984195 1loss/SRqqt2/detection_loss‚0.8168500099350084 epoch‚1 trainer/global_step‚1249 _runtime‚ 494.384477304 /loss/FRt1/assignment_loss‚3.984745115569875 1loss/FBt2/assignment_loss‚0.16053705502019966 /loss/FBt1/detection_loss‚10.586306899916508 0loss/FRt2/assignment_loss‚2.6577999071578757 2loss/SRqqt1/assignment_loss‚1.7896771533799565 1loss/FBt1/assignment_loss‚0.21335537488132883 /loss/FRt1/detection_loss‚0.5263496787299083 /loss/FRt2/detection_loss‚1.3086085120991384 0loss/SRqqt1/detection_loss‚2.391826265219422 /loss/FBt2/detection_loss‚2.0403826935243927 &loss/total_loss‚27.749231608632034 % _timestamp‚1.7845023403216054e+09 _step‚100dÅû½8R®‚(:Ç Ä°õÒ©Í…À! disk.nvme0c0n1.in‚ 45.31640625 cpu‚26.46595914158561 network.recv‚184067 disk.dm-0.in‚ 51.3046875 disk.dm-0.out‚570.7158203125 disk.dm-1.in‚0 disk.nvme0c0n1.out‚ 513.46875$ disk.nvme1n1.out‚215.28466796875 proc.cpu.threads‚17 network.sent‚ 1.398379e+06% disk./.usageGB‚1167.7499504089355 disk.md0.in‚0 disk.md0.out‚ 11.9453125 disk.nvme0n1.in‚ 45.31640625 disk.nvme0n1.out‚ 513.46875 disk.nvme1n1.in‚ 5.98828125$ memory_percent‚99.97851848602295! disk.dm-1.out‚132.25048828125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme1c1n1.in‚ 5.98828125& disk.nvme1c1n1.out‚215.28466796875& proc.memory.availableMB‚ 3.51953125 proc.memory.rssMB‚2855.5( proc.memory.percent‚17.4285888671875) disk./.usagePercent‚76.06252730232441©×úõœ¯‚(:‘ Ä°õÒúÒ·À gpu.0.gpu‚68.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚190.478' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚63.49266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ºÿaT`°‚(:Õ Ó°õÒ°Ëï¿ disk.nvme1c1n1.in‚ 5.98828125! disk.dm-0.out‚604.71630859375& disk.nvme1c1n1.out‚217.66845703125 network.sent‚ 1.449727e+06) disk./.usagePercent‚76.06252109047412$ disk./.usageGB‚1167.749855041504 disk.rbd0.out‚0 disk.md0.in‚0 disk.md0.out‚ 12.37109375 proc.cpu.threads‚17! disk.dm-1.out‚132.33642578125! disk.nvme0c0n1.in‚ 45.31640625$ disk.nvme0n1.out‚546.09521484375$ disk.nvme1n1.out‚217.66845703125$ memory_percent‚99.97799396514893& proc.memory.availableMB‚ 3.60546875 proc.memory.rssMB‚2855.5( proc.memory.percent‚17.4285888671875 disk.nvme0n1.in‚ 45.31640625 network.recv‚188409 disk.dm-1.in‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚546.09521484375 disk.nvme1n1.in‚ 5.98828125 disk.dm-0.in‚ 51.3046875 cpu‚26.488418514113377þ“›±‚(: Ó°õÒÌÄœÀ gpu.0.gpu‚46.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚201.451' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.15033333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0K@X0²+ _wandbruntime‚513  _runtime‚513¥'g‹È³Â trainer/global_step‚1299 % _timestamp‚1.7845023598577905e+09 _runtime‚ 513.920976436 _step‚101 ' lr-AdamW/pg1‚0.00011208248207316471 ' lr-AdamW/pg2‚0.00011208248207316471Ÿd Ì´Æ ' lr-AdamW/pg1‚0.00011208248207316471 ' lr-AdamW/pg2‚0.00011208248207316471 trainer/global_step‚1299 % _timestamp‚1.7845023598577905e+09 _runtime‚ 513.920976436 _step‚101e1vFÀ0µ+ _wandbruntime‚513  _runtime‚51399 ß°'¶ªO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13079902 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚50461416498688 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚49558342183936 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚50461521559040 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3302949 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565184 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 903162335744 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚49558359223296 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8976216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚50461521559040 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22838657 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚48911441074064 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 _step‚102 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9758757 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3302720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 900233669495 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13090043 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚49558342183936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48275146298880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚48275347422720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13090043 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚49558359223296 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 903162335744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 929508155904 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9758755 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13079902 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 903074314752 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9758755 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚50461416498688 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 903074314752 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 900319817961 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5673264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚47345638142976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 % _timestamp‚1.7845023598588276e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201123840 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _runtime‚ 513.92191019 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚49811657704199 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22848800 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 929684376064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚1299,õ’6'·®O @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3302720 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 900233669495 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13090043 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚49558342183936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48275146298880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚48275347422720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13090043 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚49558359223296 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 903162335744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 929508155904 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚9758755 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13079902 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 903074314752 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9758755 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚50461416498688 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 903074314752 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 900319817961 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5673264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚47345638142976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 % _timestamp‚1.7845023598588276e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201123840 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _runtime‚ 513.92191019 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚49811657704199 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22848800 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 929684376064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚1299 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13079902 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚50461416498688 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚49558342183936 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚50461521559040 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3302949 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565184 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 903162335744 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚49558359223296 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚8976216 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚50461521559040 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚22838657 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚48911441074064 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 _step‚102 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚9758757 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176220160 E=DeviceStatX»ã~sMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0fÃÚþ0¸+ _wandbruntime‚514  _runtime‚514aö„Ø»&¹µM WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚48311538426880 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚48948296102464 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 903840452608 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8982942 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22866033 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 514.187553436 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚49595708551168 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚49595708551168 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22855876 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9766078 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 % _timestamp‚1.7845023601242516e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 900909620939 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8982692 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9766076 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5677557 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚50499460975616 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22866033 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚48948313141824 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 900995769517 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 930205126656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚49849205723403 trainer/global_step‚1299 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 930387631104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚49595725590528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182504448 _step‚103 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13099955 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9766076 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚49849308911341 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47381358203904 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚50499566043136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13089800 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3305138 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207408128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 903752424448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3305385 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 903752424448 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48311745835008Ž ö¿&º¹M % _timestamp‚1.7845023601242516e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 900909620939 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8982692 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9766076 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5677557 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚50499460975616 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22866033 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚48948313141824 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 900995769517 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 930205126656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚49849205723403 trainer/global_step‚1299 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 930387631104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚49595725590528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182504448 _step‚103 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13099955 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9766076 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚49849308911341 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47381358203904 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚50499566043136 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13089800 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3305138 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207408128 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 903752424448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3305385 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 903752424448 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48311745835008 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚48311538426880 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚48948296102464 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 903840452608 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8982942 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22866033 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 514.187553436 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚49595708551168 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚49595708551168 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚22855876 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9766078 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808glQE0»+ _wandbruntime‚514  _runtime‚514:t/d¼ú 1loss/FBt1/assignment_loss‚0.16229225862171007 &loss/total_loss‚23.060847266727972 epoch‚1 % _timestamp‚1.7845023601252887e+09 /loss/FRt1/detection_loss‚0.5615542717097107 1loss/SRqqt2/detection_loss‚0.8190863042569917 -loss/FBt1/detection_loss‚5.82026750204687 trainer/global_step‚1299 2loss/SRqqt1/assignment_loss‚1.8303027555715938 0loss/FBt2/assignment_loss‚0.2954256722058377 /loss/FRt2/detection_loss‚1.3165515362606437 0loss/SRqqt1/detection_loss‚2.770424204451272 .loss/FBt2/detection_loss‚2.016210800128079 _runtime‚ 514.188351699 _step‚104 /loss/FRt1/assignment_loss‚3.943803544630084 0loss/FRt2/assignment_loss‚2.3303461280954676 1loss/SRqqt2/assignment_loss‚1.1945822887497134R5o½þ 1loss/SRqqt2/detection_loss‚0.8190863042569917 -loss/FBt1/detection_loss‚5.82026750204687 trainer/global_step‚1299 2loss/SRqqt1/assignment_loss‚1.8303027555715938 0loss/FBt2/assignment_loss‚0.2954256722058377 /loss/FRt2/detection_loss‚1.3165515362606437 0loss/SRqqt1/detection_loss‚2.770424204451272 .loss/FBt2/detection_loss‚2.016210800128079 _runtime‚ 514.188351699 _step‚104 /loss/FRt1/assignment_loss‚3.943803544630084 0loss/FRt2/assignment_loss‚2.3303461280954676 1loss/SRqqt2/assignment_loss‚1.194582288749713 1loss/FBt1/assignment_loss‚0.16229225862171007 &loss/total_loss‚23.060847266727972 epoch‚1 % _timestamp‚1.7845023601252887e+09 /loss/FRt1/detection_loss‚0.5615542717097107hP'ÅC`¾‚(:Õ â°õÒɼӿ) disk./.usagePercent‚76.06246046281525$ disk./.usageGB‚1167.748924255371& disk.nvme0c0n1.out‚552.26123046875$ memory_percent‚99.97985363006592 cpu‚26.510096941780066 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk.nvme1n1.out‚218.28271484375 disk.dm-0.in‚ 51.3046875 proc.memory.rssMB‚2855.5 proc.cpu.threads‚17& disk.nvme1c1n1.out‚218.28271484375 network.sent‚ 1.453087e+06 disk.dm-1.in‚0! disk.dm-1.out‚132.57373046875 disk.nvme0n1.in‚ 45.31640625 disk.nvme1n1.in‚ 5.98828125 disk.md0.out‚ 12.68359375& proc.memory.availableMB‚ 3.30078125! disk.nvme0c0n1.in‚ 45.31640625 disk.md0.in‚0! disk.dm-0.out‚610.59912109375( proc.memory.percent‚17.4285888671875$ disk.nvme0n1.out‚552.26123046875 disk.nvme1c1n1.in‚ 5.98828125 network.recv‚192219ÏêS{‘¿‚(:† â°õÒµç¯À gpu.0.gpu‚90.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚201.654' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚67.218 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ð0À+ _wandbruntime‚534  _runtime‚534é—sÇÁÁ ' lr-AdamW/pg2‚0.00011207309203467216 trainer/global_step‚1349 $ _timestamp‚1.784502379993779e+09 _runtime‚ 534.056940395 _step‚105 ' lr-AdamW/pg1‚0.00011207309203467216rK5ÉËÂÅ ' lr-AdamW/pg1‚0.00011207309203467216 ' lr-AdamW/pg2‚0.00011207309203467216 trainer/global_step‚1349 $ _timestamp‚1.784502379993779e+09 _runtime‚ 534.056940395 _step‚105i£{§0Ã+ _wandbruntime‚534  _runtime‚534ó³_$įO ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 937100077568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51426660545536 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13576654 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚51689193755433 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚50755027422704 _step‚106 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 964411604992 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9313481 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3425717 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13586795 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 934149293369 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10124807 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49130420909056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52363672602112 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50095023414272 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 937012056576 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7845023799947803e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚50755044462064 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 trainer/global_step‚1349 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51426677584896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51426660545536 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 190900224 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 _runtime‚ 534.057909617 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 937012056576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 964602505216 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13586795 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚52363777662464 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚52363777662464 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 937100077568 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚52363672602112 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23711602 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes._cZVlarge_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215803904 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51426677584896 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9313749 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 934063144903aðé¹'ųO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51426677584896 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9313749 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 934063144903 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 937100077568 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚51426660545536 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13576654 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚51689193755433 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚50755027422704 _step‚106 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 964411604992 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9313481 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3425717 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13586795 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 934149293369 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10124807 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚49130420909056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52363672602112 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚50095023414272 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 937012056576 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7845023799947803e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚50755044462064 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 trainer/global_step‚1349 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51426677584896 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚51426660545536 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 190900224 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 _runtime‚ 534.057909617 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 937012056576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 964602505216 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚13586795 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚52363777662464 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚52363777662464 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 937100077568 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚52363672602112 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚23711602 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215803904 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712j>—°0Æ+ _wandbruntime‚534  _runtime‚534z~g°¿&ǹM ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50791916529824 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚52401822146560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10132126 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50131410259456 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13596707 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23728835 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10132128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _runtime‚ 534.322186483 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51464026912768 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5892054 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13586552 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3428116 trainer/global_step‚1349 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10132126 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180407296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51726638586811 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 934825244925 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10132128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9320421 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7845023802588599e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9320170 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 937778194432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 _step‚107 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13586552 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 937690166272 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 965294193152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 934739096347 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23728835 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205310976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0â¿[ Ã&ȽM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51464043952128 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚52401822146560 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50791916529824 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚52401822146560 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10132126 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚50131410259456 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚13596707 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚23728835 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10132128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _runtime‚ 534.322186483 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚51464026912768 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5892054 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚13586552 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3428116 trainer/global_step‚1349 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10132126 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180407296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51726638586811 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 934825244925 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10132128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9320421 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7845023802588599e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9320170 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 937778194432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 _step‚107 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13586552 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 937690166272 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 965294193152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 934739096347 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚23728835 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205310976 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14073406 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚53566626618841 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚54265928705536 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 971037819392 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52598630810704 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚51914525649408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚50915178771456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 999371781632 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54265928705536 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14083547 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 _runtime‚ 554.361795479 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14073406 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚53566626618841 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚54265928705536 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 971037819392 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚52598630810704 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚51914525649408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚50915178771456 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 999371781632 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54265928705536 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14083547 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 _runtime‚ 554.361795479 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9656808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9657047 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 _runtime‚ 554.630834529 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚54303973182464 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3550254 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53332345274368 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1399 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 971715936256 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚52635502878464 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6106554 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3550490 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 968654720333 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24581480 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24591637 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚50950849025024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 968568571755 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0½ Ê3Ö¿M ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53332345274368 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚1399 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 971715936256 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚52635502878464 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6106554 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3550490 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 968654720333 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24581480 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24591637 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚50950849025024 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 968568571755 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9656808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9657047 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 _runtime‚ 554.630834529 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚54303973182464 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3550254 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239o0¦²*0×+ _wandbruntime‚554  _runtime‚554fuQÍØú 1loss/SRqqt2/assignment_loss‚1.239574896770562 0loss/FBt2/assignment_loss‚0.2384668260556022 /loss/FRt2/detection_loss‚1.3837197495542557 .loss/FBt1/detection_loss‚4.750891069199669 epoch‚1 _step‚112 2loss/SRqqt1/assignment_loss‚1.7576808271597522 /loss/FBt2/detection_loss‚3.2269288729518646 &loss/total_loss‚23.935916857710104 _runtime‚ 554.63141617 0loss/FRt2/assignment_loss‚2.4672610845133573 0loss/SRqqt1/detection_loss‚3.401581698716279 /loss/FRt1/assignment_loss‚3.975336314451815 1loss/FBt1/assignment_loss‚0.16672610445590788 /loss/FRt1/detection_loss‚0.5566264877468142 1loss/SRqqt2/detection_loss‚0.7711229261342261 trainer/global_step‚1399 $ _timestamp‚1.784502400567896e+09r±RãÙþ 1loss/SRqqt2/detection_loss‚0.7711229261342261 trainer/global_step‚1399 $ _timestamp‚1.784502400567896e+09 1loss/SRqqt2/assignment_loss‚1.239574896770562 0loss/FBt2/assignment_loss‚0.2384668260556022 /loss/FRt2/detection_loss‚1.3837197495542557 .loss/FBt1/detection_loss‚4.750891069199669 epoch‚1 _step‚112 2loss/SRqqt1/assignment_loss‚1.7576808271597522 /loss/FBt2/detection_loss‚3.2269288729518646 &loss/total_loss‚23.935916857710104 _runtime‚ 554.63141617 0loss/FRt2/assignment_loss‚2.4672610845133573 0loss/SRqqt1/detection_loss‚3.401581698716279 /loss/FRt1/assignment_loss‚3.975336314451815 1loss/FBt1/assignment_loss‚0.16672610445590788 /loss/FRt1/detection_loss‚0.5566264877468142p?]$ÓYÚ‚(:Î €±õÒò¨À) disk./.usagePercent‚76.07797741637597# disk.nvme1n1.out‚281.3564453125 disk.dm-0.out‚866.1650390625$ memory_percent‚99.98314380645752 proc.memory.rssMB‚2855.5 proc.cpu.threads‚17 network.recv‚201580 disk.nvme1n1.in‚ 5.98828125 disk.md0.in‚0 disk.md0.out‚ 13.44140625 disk.dm-0.in‚ 51.3046875 disk.dm-1.out‚136.4267578125% disk.nvme0c0n1.out‚750.2431640625 disk.nvme1c1n1.in‚ 5.98828125 network.sent‚ 1.591719e+06$ disk./.usageGB‚1167.987148284912 disk.rbd0.out‚0 disk.nvme0n1.in‚ 45.31640625# disk.nvme0n1.out‚750.2431640625& proc.memory.availableMB‚ 2.76171875( proc.memory.percent‚17.4285888671875 disk.dm-1.in‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 45.31640625% disk.nvme1c1n1.out‚281.3564453125 cpu‚26.54905695012619´´ð¼›Û‚(: €±õÒ‰þ¿À gpu.0.gpu‚57.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚201.777' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.25899999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã¹ÿÛXÜ‚(:Í ±õÒÊðÌ¿& proc.memory.availableMB‚ 2.96484375( proc.memory.percent‚17.4285888671875 network.sent‚ 1.640334e+06# disk.nvme0n1.out‚784.0791015625 disk.nvme1n1.in‚ 5.98828125$ memory_percent‚99.98190402984619 cpu‚26.56710909472078 proc.cpu.threads‚17% disk.nvme1c1n1.out‚283.4482421875 disk.md0.out‚ 13.7421875 disk.dm-0.in‚ 51.3046875 disk.dm-1.in‚0 disk.dm-1.out‚136.4970703125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.md0.in‚0 network.recv‚205946) disk./.usagePercent‚75.88548509640951$ disk./.usageGB‚1165.031909942627 disk.nvme0n1.in‚ 45.31640625# disk.nvme1n1.out‚283.4482421875 proc.memory.rssMB‚2855.5 disk.nvme1c1n1.in‚ 5.98828125 disk.dm-0.out‚901.3779296875! disk.nvme0c0n1.in‚ 45.31640625% disk.nvme0c0n1.out‚784.07910156256&·®œÝ‚(:‘ ±õÒɨˆÀ gpu.0.gpu‚49.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚123.012' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚41.004000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñîF0Þ+ _wandbruntime‚574  _runtime‚574›þ˜oÇßÁ ' lr-AdamW/pg1‚0.00011205162080094316 ' lr-AdamW/pg2‚0.00011205162080094316 trainer/global_step‚1449 % _timestamp‚1.7845024207778292e+09 _runtime‚ 574.84101121 _step‚113Ÿ)ToËàÅ ' lr-AdamW/pg2‚0.00011205162080094316 trainer/global_step‚1449 % _timestamp‚1.7845024207778292e+09 _runtime‚ 574.84101121 _step‚113 ' lr-AdamW/pg1‚0.00011205162080094316qtC0á+ _wandbruntime‚574  _runtime‚574µ†íÔ½'â·O ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1004975561216 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚54442234198704 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3670490 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207415296 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1004887540224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1001722095719 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚53734451400704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6316764 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚55444059482249 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54442251238064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚55163297268736 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚56168289869312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚52699911730176 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7845024207788029e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14580299 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182511616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9987496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10856905 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚52699936633856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14580299 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10856907 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚55163314308096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚55443956294423 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 574.841952003 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9987254 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚56168184808960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10856907 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25427063 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992ˆÚA"ã»O M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚25437206 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3670729 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _step‚114 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚56168289869312 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚14570158 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚55163314308096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1001808244185 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1034332255232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚1449 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10856905 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10856905 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚52699936633856 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚14580299 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10856907 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚55163314308096 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚55443956294423 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 574.841952003 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚9987254 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚56168184808960 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10856907 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚25427063 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1004975561216 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚54442234198704 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565184 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3670490 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207415296 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1004887540224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1001722095719 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚53734451400704 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6316764 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚55444059482249 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚54442251238064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚55163297268736 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚56168289869312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚52699911730176 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7845024207788029e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14580299 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182511616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9987496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1002484195741 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚55200680675328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1005565649920 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25444282 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚53770846933504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1005653678080 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1035032637952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚54479106266464 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25444282 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10864228 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9993946 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25454439 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14590211 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14590211 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚55200680675328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚54479123305824 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚56206334353408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10864226 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1002398047163 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚56206229285888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10864226 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1005653678080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55481504313627 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 $ _timestamp‚1.784502421047986e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚55200663635968 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚56206334353408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25454439 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6321057 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10864228 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6321054 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3672892 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578ÿÏùpÉ&æÃM VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚56206334353408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10864226 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1002398047163 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚56206229285888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10864226 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1005653678080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55481504313627 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 $ _timestamp‚1.784502421047986e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚55200663635968 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚56206334353408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚25454439 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6321057 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10864228 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6321054 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3672892 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚52735631791104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14580056 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚55481607501565 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1002484195741 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚55200680675328 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1005565649920 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚25444282 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚53770846933504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1005653678080 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1035032637952 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚54479106266464 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚25444282 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10864228 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9993946 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚25454439 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14590211 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14590211 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚55200680675328 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12301 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚54479123305824sÏùÁ0ç+ _wandbruntime‚575  _runtime‚575ú}º-èú %loss/total_loss‚26.29795596381561 2loss/SRqqt2/assignment_loss‚1.2705189695402694 1loss/SRqqt1/detection_loss‚2.1082116625403264 /loss/FBt1/assignment_loss‚0.377779153378334 1loss/FBt2/assignment_loss‚0.10565649839391507 epoch‚1 % _timestamp‚1.7845024210490286e+09 _runtime‚ 575.112760985 /loss/FRt1/assignment_loss‚3.921758097440458 2loss/SRqqt1/assignment_loss‚1.7866341535288555 .loss/FRt1/detection_loss‚0.654604681127239 /loss/FRt2/detection_loss‚1.4436982083159398 .loss/FBt1/detection_loss‚7.114197482089294 .loss/FBt2/detection_loss‚4.254382864244013 trainer/global_step‚1449 _step‚116 0loss/FRt2/assignment_loss‚2.3859546377326333 1loss/SRqqt2/detection_loss‚0.8745595554843301òVBéþ %loss/total_loss‚26.29795596381561 2loss/SRqqt2/assignment_loss‚1.2705189695402694 1loss/SRqqt1/detection_loss‚2.1082116625403264 /loss/FBt1/assignment_loss‚0.377779153378334 1loss/FBt2/assignment_loss‚0.10565649839391507 epoch‚1 % _timestamp‚1.7845024210490286e+09 _runtime‚ 575.112760985 /loss/FRt1/assignment_loss‚3.921758097440458 2loss/SRqqt1/assignment_loss‚1.7866341535288555 .loss/FRt1/detection_loss‚0.654604681127239 /loss/FRt2/detection_loss‚1.4436982083159398 .loss/FBt1/detection_loss‚7.114197482089294 .loss/FBt2/detection_loss‚4.254382864244013 trainer/global_step‚1449 _step‚116 0loss/FRt2/assignment_loss‚2.3859546377326333 1loss/SRqqt2/detection_loss‚0.8745595554843301tþžhSê‚(:È ž±õÒ“ ý¿ network.sent‚ 1.659442e+06 disk.md0.out‚14.09375 disk.dm-0.in‚ 53.19140625! disk.dm-0.out‚904.50927734375 proc.memory.rssMB‚2855.5 network.recv‚209829) disk./.usagePercent‚75.88549876248015# disk.nvme1c1n1.out‚ 284.06640625! disk.nvme1n1.out‚ 284.06640625 disk.dm-1.out‚136.6533203125 disk.rbd0.out‚0 disk.nvme0n1.in‚ 47.203125( proc.memory.percent‚17.4285888671875% disk./.usageGB‚1165.0321197509766 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 47.203125& disk.nvme0c0n1.out‚787.52392578125$ disk.nvme0n1.out‚787.52392578125& proc.memory.availableMB‚ 2.85546875 cpu‚26.58166914371435 disk.nvme1c1n1.in‚ 5.98828125 disk.nvme1n1.in‚ 5.98828125 disk.md0.in‚0 disk.dm-1.in‚0$ memory_percent‚99.98257160186768 proc.cpu.threads‚17¶GÐë‚(:… ž±õÒ½ð¤À gpu.0.gpu‚15.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.393' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.131 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0#ÕÐ0ì+ _wandbruntime‚595  _runtime‚595š¹@=Èí trainer/global_step‚1499 % _timestamp‚1.7845024411678164e+09 _runtime‚ 595.230943691 _step‚117 ' lr-AdamW/pg1‚0.00011203953994961212 ' lr-AdamW/pg2‚0.00011203953994961212­œ¹qÌîÆ % _timestamp‚1.7845024411678164e+09 _runtime‚ 595.230943691 _step‚117 ' lr-AdamW/pg1‚0.00011203953994961212 ' lr-AdamW/pg2‚0.00011203953994961212 trainer/global_step‚1499uŒÈk0ï+ _wandbruntime‚595  _runtime‚595ñf»ÿUð¹O SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚58070440912384 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚58070545972736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15066910 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚57321492345657 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚58070440912384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11222955 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 trainer/global_step‚1499 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚57031632669696 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15066910 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56285854626064 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚57031615630336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚57031632669696 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1038913303040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7845024411685336e+09 _step‚118 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182511616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1035637719593 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15077051 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11222955 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1038825282048 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54484694496256 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1038825282048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15077051 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10324997 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26300008 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 [HDeviceStatsMonitor.on_train_batch_start/requested_by’§åjtes.large_pool.freed‚56285837586704 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26289865 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11222957 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1038913303040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1069284614656 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11222955 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 trainer/global_step‚1499 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚57031632669696 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15066910 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159995904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56285854626064 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚57031615630336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚57031632669696 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1038913303040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 % _timestamp‚1.7845024411685336e+09 _step‚118 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182511616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1035637719593 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15077051 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11222955 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565184 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1038825282048 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚54484694496256 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1038825282048 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚15077051 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10324997 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚26300008 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚56285837586704 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚26289865 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11222957 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1038913303040 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1069284614656 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚58070440912384 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚58070545972736 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15066910 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚57321492345657 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚58070440912384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11230276 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11230278 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1036227522571 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15086963 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55590552596480 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚57358937177035 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚58108485389312 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1039591419904 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26307084 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6535554 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 595.500278726 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚119 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54520389653504 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10331459 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚57068999036928 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3796153 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚57359040364973 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 trainer/global_step‚1499 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55590343091200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26317241 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3795905 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1070162942976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1039591419904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚57068981997568éž Í&ôÇM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1039591419904 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚26307084 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6535554 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚ 595.500278726 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚119 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159995904 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54520389653504 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10331459 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148578 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚57068999036928 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3796153 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚57359040364973 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 trainer/global_step‚1499 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55590343091200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26317241 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3795905 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1070162942976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1039591419904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚57068981997568 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚57068981997568 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11230276 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11230278 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1036227522571 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15086963 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55590552596480 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚57358937177035 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚58108485389312 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12372 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584wù me0õ+ _wandbruntime‚595  _runtime‚595ñÒp¬öü /loss/FRt1/detection_loss‚0.6372788777412214 /loss/FBt1/detection_loss‚2.9298197221581246 .loss/FBt2/detection_loss‚2.118286479814351 %loss/total_loss‚19.81742844801753 0loss/FRt1/assignment_loss‚3.7565856352830544 /loss/FRt2/detection_loss‚1.2796675948198573 epoch‚1 _runtime‚ 595.500887797 2loss/SRqqt2/assignment_loss‚1.1946863711152953 1loss/FBt1/assignment_loss‚0.26722395966¶(z¾V219037 0loss/FBt2/assignment_loss‚0.2713020161155209 1loss/SRqqt1/detection_loss‚2.6285979111551594 trainer/global_step‚1499 % _timestamp‚1.7845024414379997e+09 _step‚120 /loss/FRt2/assignment_loss‚2.298989211144404 1loss/SRqqt1/assignment_loss‚1.574300322526445 1loss/SRqqt2/detection_loss‚0.8606903464819077!H÷€ %loss/total_loss‚19.81742844801753 0loss/FRt1/assignment_loss‚3.7565856352830544 /loss/FRt2/detection_loss‚1.2796675948198573 epoch‚1 _runtime‚ 595.500887797 2loss/SRqqt2/assignment_loss‚1.1946863711152953 1loss/FBt1/assignment_loss‚0.26722395966219037 0loss/FBt2/assignment_loss‚0.2713020161155209 1loss/SRqqt1/detection_loss‚2.6285979111551594 trainer/global_step‚1499 % _timestamp‚1.7845024414379997e+09 _step‚120 /loss/FRt2/assignment_loss‚2.298989211144404 1loss/SRqqt1/assignment_loss‚1.574300322526445 1loss/SRqqt2/detection_loss‚0.8606903464819077 /loss/FRt1/detection_loss‚0.6372788777412214 /loss/FBt1/detection_loss‚2.9298197221581246 .loss/FBt2/detection_loss‚2.118286479814351x1’øNø‚(:à ­±õÒŠˆŸÀ disk.dm-1.out‚136.7236328125% disk.nvme0c0n1.out‚800.9521484375 cpu‚26.588607044324267 network.sent‚ 1.721441e+06! disk.nvme1n1.out‚ 287.71484375 disk.md0.in‚0 disk.md0.out‚14.34375 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚75.88549776858412 disk.nvme1c1n1.in‚ 5.98828125 disk.nvme1n1.in‚ 5.98828125 disk.dm-0.in‚ 53.19140625 disk.nvme0n1.in‚ 47.203125# disk.nvme0n1.out‚800.9521484375( proc.memory.percent‚17.4285888671875% disk./.usageGB‚1165.0321044921875 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 47.203125$ memory_percent‚99.98295307159424& proc.memory.availableMB‚ 2.79296875 proc.memory.rssMB‚2855.5 network.recv‚215676# disk.nvme1c1n1.out‚ 287.71484375 disk.dm-0.out‚ 920.9609375‡²Ä‹ù‚(:… ­±õÒ¬ðÆÀ gpu.0.gpu‚63.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚196.599' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚65.533 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ç‘æÑ0ú+ _wandbruntime‚614  _runtime‚614MMœèÈû ' lr-AdamW/pg1‚0.00011202656252851468 ' lr-AdamW/pg2‚0.00011202656252851468 trainer/global_step‚1549 % _timestamp‚1.7845024606601164e+09 _runtime‚ 614.724598103 _step‚1218Ló›ÌüÆ ' lr-AdamW/pg1‚0.00011202656252851468 ' lr-AdamW/pg2‚0.00011202656252851468 trainer/global_step‚1549 % _timestamp‚1.7845024606601164e+09 _runtime‚ 614.724598103 _step‚121y§ø0ý+ _wandbruntime‚614  _runtime‚614˜ZSI¿'þ¹O LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _runtime‚ 614.724662683 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59972802076160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚122 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15563662 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58899933991936 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59972802076160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚1549 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚56269427454976 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11589007 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚58129440974704 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1069467195001 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10662763 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6745764 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚58129458014064 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚57373885928448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1104253155328 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59972697015808 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚58899951031296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15563662 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3916751 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59972697015808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1072851044864 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6745767 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10662515 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59198925209065 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1072851044864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚58899933991936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15573803 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58899951031296 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27162810 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205318144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ùͧÃ'ÿ½O L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚15563662 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58899933991936 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59972802076160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚1549 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚56269427454976 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88020992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11589007 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187826 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚58129440974704 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1069467195001 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12372 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10662763 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6745764 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148466 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚58129458014064 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚57373885928448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1104253155328 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59972697015808 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚58899951031296 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15563662 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3916751 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚59972697015808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1072851044864 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6745767 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181447 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10141 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10662515 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59198925209065 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682743 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1072851044864 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚58899933991936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15573803 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58899951031296 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27162810 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205318144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _runtime‚ 614.724662683 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59972802076160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _step‚122zN›eõ0€+ _wandbruntime‚614  _runtime‚614NÄ{Ì$ÃM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27180043 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10669301 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59236370040443 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚58166313042464 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1073441133568 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15573560 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1073529161728 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3919247 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚57410277361152 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58166330081824 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚60010741492736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚58937317398528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7845024609279654e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27180043 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11596326 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1073529161728 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11596326 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6750054 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1070143146557 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3919501 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904°m§0Í&‚ÇM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11596326 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1073529161728 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚11596326 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6750054 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1070143146557 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3919501 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27180043 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10669301 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚59236370040443 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚58166313042464 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1073441133568 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15573560 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1073529161728 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3919247 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚57410277361152 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58166330081824 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚60010741492736 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚58937317398528 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 % _timestamp‚1.7845024609279654e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27180043 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712{á†îN0ƒ+ _wandbruntime‚614  _runtime‚614,c;„ý 0loss/FRt2/assignment_loss‚2.3638419973740885 1loss/FBt1/assignment_loss‚0.13603275298417453 /loss/FRt2/detection_loss‚1.3444987480135573 &loss/total_loss‚23.412930499872232 _step‚124 0loss/FRt1/assignment_loss‚3.7221777230193935 0loss/FBt2/assignment_loss‚0.0641623216059678 .loss/FBt1/detection_loss‚7.136390034476395 2loss/SRqqt1/assignment_loss‚1.4502584850568263 2loss/SRqqt2/assignment_loss‚1.2637828395254147 0loss/SRqqt1/detection_loss‚2.165282155140068 1loss/SRqqt2/detection_loss‚0.8175697999340732 .loss/FBt2/detection_loss‚2.435803641638388 trainer/global_step‚1549 /loss/FRt1/detection_loss‚0.5131300011038847 epoch‚1 % _timestamp‚1.7845024609290097e+09 _runtime‚ 614.991883687ê<ø… .loss/FBt1/detection_loss‚7.136390034476395 2loss/SRqqt1/assignment_loss‚1.4502584850568263 2loss/SRqqt2/assignment_loss‚1.2637828395254147 0loss/SRqqt1/detection_loss‚2.165282155140068 1loss/SRqqt2/detection_loss‚0.8175697999340732 .loss/FBt2/detection_loss‚2.435803641638388 trainer/global_step‚1549 /loss/FRt1/detection_loss‚0.5131300011038847 epoch‚1 % _timestamp‚1.7845024609290097e+09 _runtime‚ 614.991883687 0loss/FRt2/assignment_loss‚2.3638419973740885 1loss/FBt1/assignment_loss‚0.13603275298417453 /loss/FRt2/detection_loss‚1.3444987480135573 &loss/total_loss‚23.412930499872232 _step‚124 0loss/FRt1/assignment_loss‚3.7221777230193935 0loss/FBt2/assignment_loss‚0.0641623216059678|×'Ä„[†‚(:Ð ¼±õÒÑ›¿¿ disk.md0.in‚0! disk.dm-0.out‚928.54736328125! disk.dm-1.out‚137.05029296875 disk.nvme0c0n1.in‚ 47.203125$ disk.nvme0n1.out‚808.79931640625$ disk.nvme1n1.out‚288.76318359375 disk.dm-1.in‚0 disk.rbd0.out‚0 network.sent‚ 1.771148e+06 network.recv‚220861& disk.nvme0c0n1.out‚808.79931640625 disk.nvme1c1n1.in‚ 5.98828125& disk.nvme1c1n1.out‚288.76318359375$ memory_percent‚99.98464584350586( proc.memory.percent‚17.4285888671875 proc.cpu.threads‚17 disk.nvme0n1.in‚ 47.203125 disk.nvme1n1.in‚ 5.98828125 disk.dm-0.in‚ 53.19140625 disk.rbd0.in‚0$ proc.memory.availableMB‚2.515625 proc.memory.rssMB‚2855.5 cpu‚26.573048224195368 disk.md0.out‚ 14.8046875) disk./.usagePercent‚75.70850302120546% disk./.usageGB‚1162.3147926330566µSnÇœ‡‚(:‘ ¼±õÒ¢ç€À gpu.0.gpu‚54.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚163.466' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚54.488666666666674 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î{¿Wˆ‚(:Ì ˱õÒ¥±û¿ network.recv‚225298 disk.dm-0.in‚ 53.19140625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 47.203125$ memory_percent‚99.98359680175781" proc.memory.availableMB‚2.6875 proc.memory.rssMB‚2855.5 proc.cpu.threads‚17) disk./.usagePercent‚75.71034818921682$ disk.nvme0n1.out‚821.95458984375& disk.nvme1c1n1.out‚289.22900390625( proc.memory.percent‚17.4285888671875 cpu‚26.558108674482042! disk.dm-0.out‚941.71728515625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 47.203125 disk.nvme1c1n1.in‚ 5.98828125 disk.md0.in‚0 network.sent‚ 1.835009e+06% disk./.usageGB‚1162.3431205749512! disk.dm-1.out‚137.14794921875 disk.rbd0.out‚0& disk.nvme0c0n1.out‚821.95458984375 disk.nvme1n1.in‚ 5.98828125$ disk.nvme1n1.out‚289.22900390625 disk.md0.out‚14.96875BW~뛉‚(: ˱õÒε¥À gpu.0.gpu‚51.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚203.045' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.68166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÎE"0Š+ _wandbruntime‚634  _runtime‚634Ok±ÔÈ‹Â ' lr-AdamW/pg1‚0.00011201268874551061 ' lr-AdamW/pg2‚0.00011201268874551061 trainer/global_step‚1599 % _timestamp‚1.7845024804202034e+09 _runtime‚ 634.484624348 _step‚125U†´âÌŒÆ _step‚125 ' lr-AdamW/pg1‚0.00011201268874551061 ' lr-AdamW/pg2‚0.00011201268874551061 trainer/global_step‚1599 % _timestamp‚1.7845024804202034e+09 _runtime‚ 634.484624348}Zø[ç0+ _wandbruntime‚634  _runtime‚634td¿'Ž¹O [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚60768252353536 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6960267 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4040062 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11955055 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10141 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12372 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12301 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 _step‚126 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚28025612 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188803072 _runtime‚ 634.484725918 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16070555 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚61875058179584 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1106788786688 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58054210221056 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚59973061402064 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16070555 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1103296670409 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11955057 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚60768269392896 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚59193401263616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28015469 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚61076358072473 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚59973044362704 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11955057 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚60768269392896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚1599 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61874953119232 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚61874953119232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1106788786688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1106700765696 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11000574 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1139404749312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1106700765696 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12301 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88020992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188803072 _runtime‚ 634.484725918 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16070555 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚61875058179584 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1106788786688 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚58054210221056 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚59973061402064 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16070555 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1103296670409 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11955057 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚60768269392896 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚59193401263616 LFDeviceStatsMonitor.on_train_batch_startÿN˜Ì /oversize_allocations.allocated‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚28015469 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10143 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚61076358072473 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565184 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚59973044362704 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11955057 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚60768269392896 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚1599 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61874953119232 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚61874953119232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1106788786688 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1106700765696 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11000574 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1139404749312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10143 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159995904 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1106700765696 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11007055 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1140100078592 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16070312 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚60009916430464 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16080467 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚1599 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚60805618720768 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚61913102663680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1107378875392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6964557 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186698752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚60805635760128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28042845 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚59229793854464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1139913379840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚60009933469824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1103972621965 % _timestamp‚1.7845024806877031e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 _runtime‚ 634.750955944 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4042760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚60805618720768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211602432 _step‚127 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61912997596160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11007317 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743&3pGÍ&’ÇM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12372 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10155 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4042760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10157 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚60805618720768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211602432 _step‚127 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61912997596160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11007317 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682743 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1107378875392 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚28032688 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚61913102663680 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚28032688 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105067520 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565184 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚60805635760128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11007055 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1140100078592 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16070312 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159995904 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚60009916430464 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16080467 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181447 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚1599 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚60805618720768 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565184 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚61913102663680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1107378875392 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028160 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6964557 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105067520 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186698752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚60805635760128 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10155 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚28042845 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚59229793854464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1139913379840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚60009933469824 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187938 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1103972621965 % _timestamp‚1.7845024806877031e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 _runtime‚ 634.750955944 E&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.27501118183135986 particle/accuracy_1_of_3‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9788332581520081 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7737799882888794 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5116642713546753 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.2857142984867096 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.058738239109516144 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.17000000178813934 +jet/accuracy_2_of_4‚0.20717914402484894 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6036494374275208 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5173217058181763 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.0911451056599617 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.28603899478912354 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.34377118945121765 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4390813410282135 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.20622478425502777 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4775494635105133 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.27554601430892944 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0008503401768393815 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5441441535949707 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.699999988079071 jet/accuracy_4_of_5‚0 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4689983129501343 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.8917456865310669 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.06683679670095444 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.08268766105175018 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 particle/accuracy_2_of_4‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.44277313351631165 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.43471458554267883 6 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.956798255443573 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.1428571492433548 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9743523597717285 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.04982250556349754 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9780058264732361 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.24529717862606049 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.35133472084999084 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 -jet/accuracy_4_of_4‚0.0059523810632526875 particle/accuracy_1_of_1‚1 +particle/sensitivity‚0.6028147339820862 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9966152906417847 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9807692170143127 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.30648404359817505 *particle/specificity‚0.869841456413269 particle/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.18370364606380463 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9902777671813965 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.049326080828905106 particle/accuracy_1_of_2‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_2_of_2‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5093101263046265 trainer/global_step‚1601 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.055195003747940063 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.32075679302215576 *jet/accuracy_1_of_2‚0.6486139297485352 particle/accuracy_4_of_5‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1309555023908615 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9750211834907532 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9764621257781982 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.22208388149738312 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7200000286102295 ,jet/accuracy_3_of_3‚0.029538992792367935 jet/accuracy_2_of_5‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.14714768528938293 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.013598017394542694 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3450619578361511 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.0811813697218895 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.47829487919807434 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4537234604358673 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.010416666977107525 5 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.50056391954422 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 jet/accuracy_3_of_5‚0 _runtime‚ 651.564347728 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2992420792579651 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3422773480415344 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8363640904426575 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.042118657380342484 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2808641791343689 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4766712188720703 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.285714298-ÀÍ~"4867096 *loss/val_total_loss‚21.814864801346204 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.48759809136390686 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9961563944816589 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.08562818914651871 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.24166667461395264 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.27054569125175476 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_2_of_3‚0.25392764806747437 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.21883103251457214 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.6723914742469788 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.002805144991725683 _step‚129 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.08244830369949341 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9717614054679871 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 /particle/accuracy_3_of_3‚0.9984782934188843 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.3723684251308441 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.7321428656578064 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.06541433930397034 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.05764123424887657 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9867523908615112 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.132505863904953 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.26893940567970276 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5704711079597473 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.25 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9901608228683472 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9512710571289062 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4254513382911682 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2954545319080353 *jet/accuracy_2_of_2‚0.0736984834074974 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.2857142984867096 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.001604903256520629 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0005537098622880876 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7200000286102295 (particle/accuracy‚0.8125077486038208 % _timestamp‚1.7845024975007615e+09 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.0564107820391655 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9880725145339966 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9901608228683472 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.35428085923194885 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5162296891212463 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.09044890850782394 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.808515191078186 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.12873679399490356 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.009988637641072273 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9316734075546265 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.2165369689464569 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.18337252736091614 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9389287829399109 epoch‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9952380657196045 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.18463541567325592 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_5‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9855572581291199 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9909039735794067 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.17469485104084015 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.557227373123169 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1433468461036682 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9888794422149658 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.044092077761888504 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5298245549201965 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5681947469711304 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1416352540254593 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.31894487142562866 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14358744025230408 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08939094096422195 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.34474095702171326 +jet/accuracy_3_of_4‚0.02465127408504486 /particle/accuracy_4_of_4‚0.9654484987258911 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.33974358439445496 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.945872962474823 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.47832366824150085 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5187909007072449 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.7916666269302368 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 particle/accuracy_2_of_5‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.0645403042435646 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9898989200592041 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5155471563339233 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.47832366824150085 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.34133094549179077 )jet/accuracy_1_of_3‚0.743854820728302 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9914332032203674 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1369004249572754 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.09373755007982254 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.30648404359817505 6validation_average_jet_accuracy‚0.2558794915676117 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.125 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17323750257492065 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5053479671478271 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5167708992958069 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.24166667461395264 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7103351950645447 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.05766867846250534 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9839357733726501 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1363593190908432 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08939094096422195 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5452991127967834 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02083333395421505 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7600334882736206 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.46506017446517944 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14063718914985657 &particle/f_score‚0.579971969127655 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.05764123424887657 /particle/accuracy_3_of_4‚0.9970238208770752 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172È„<<:>š´| 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2808641791343689 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.4766712188720703 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.2857142984867096 *loss/val_total_loss‚21.814864801346204 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.48759809136390686 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9961563944816589 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.08562818914651871 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.24166667461395264 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.27054569125175476 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_2_of_3‚0.25392764806747437 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.21883103251457214 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.6723914742469788 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.002805144991725683 _step‚129 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.08244830369949341 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9717614054679871 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 /particle/accuracy_3_of_3‚0.9984782934188843 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.3723684251308441 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.7321428656578064 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.06541433930397034 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.05764123424887657 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9867523908615112 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.1666666716337204 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.132505863904953 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.26893940567970276 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5704711079597473 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.25 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9901608228683472 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9512710571289062 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4254513382911682 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2954545319080353 *jet/accuracy_2_of_2‚0.0736984834074974 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.2857142984867096 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.001604903256520629 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0005537098622880876 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7200000286102295 (particle/accuracy‚0.8125077486038208 % _timestamp‚1.7845024975007615e+09 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.0564107820391655 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9880725145339966 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9901608228683472 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.35428085923194885 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5162296891212463 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.09044890850782394 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.808515191078186 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.12873679399490356 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.009988637641072273 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9316734075546265 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.2165369689464569 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.18337252736091614 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9389287829399109 epoch‚1 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9952380657196045 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.18463541567325592 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 particle/accuracy_1_of_5‚1 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9855572581291199 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9909039735794067 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.17469485104084015 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.557227373123169 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1433468461036682 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9888794422149658 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.044092077761888504 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5298245549201965 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5681947469711304 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1416352540254593 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.31894487142562866 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14358744025230408 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08939094096422195 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.34474095702171326 +jet/accuracy_3_of_4‚0.02465127408504486 /particle/accuracy_4_of_4‚0.9654484987258911 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.33974358439445496 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.945872962474823 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.47832366824150085 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5187909007072449 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.7916666269302368 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 particle/accuracy_2_of_5‚1 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.0645403042435646 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9898989200592041 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5155471563339233 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.47832366824150085 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.34133094549179077 )jet/accuracy_1_of_3‚0.743854820728302 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9914332032203674 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1369004249572754 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.09373755007982254 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.30648404359817505 6validation_average_jet_accuracy‚0.2558794915676117 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.125 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.17323750257492065 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5053479671478271 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5167708992958069 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.24166667461395264 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7103351950645447 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.05766867846250534 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9839357733726501 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1363593190908432 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08939094096422195 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.5452991127967834 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.02083333395421505 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7600334882736206 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.46506017446517944 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14063718914985657 &particle/f_score‚0.579971969127655 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.05764123424887657 /particle/accuracy_3_of_4‚0.9970238208770752 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9547009468078613 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 6 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.969062089920044 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.45697101950645447 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.272827684879303 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.45516055822372437 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.27501118183135986 particle/accuracy_1_of_3‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9788332581520081 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7737799882888794 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5116642713546753 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.2857142984867096 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.058738239109516144 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.17000000178813934 +jet/accuracy_2_of_4‚0.20717914402484894 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6036494374275208 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5173217058181763 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.0911451056599617 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.28603899478912354 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.34377118945121765 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4390813410282135 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.20622478425502777 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4775494635105133 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.27554601430892944 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0008503401768393815 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5441441535949707 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.699999988079071 jet/accuracy_4_of_5‚0 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4689983129501343 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.8917456865310669 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.06683679670095444 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.08268766105175018 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 particle/accuracy_2_of_4‚1 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.44277313351631165 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.43471458554267883 6 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.956798255443573 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.1428571492433548 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9743523597717285 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.04982250556349754 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9780058264732361 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.24529717862606049 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.35133472084999084 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 -jet/accuracy_4_of_4‚0.0059523810632526875 particle/accuracy_1_of_1‚1 +particle/sensitivity‚0.6028147339820862 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9966152906417847 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9807692170143127 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.30648404359817505 *particle/specificity‚0.869841456413269 particle/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.18370364606380463 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9902777671813965 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.049326080828905106 particle/accuracy_1_of_2‚1 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_2_of_2‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.5093101263046265 trainer/global_step‚1601 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.055195003747940063 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.32075679302215576 *jet/accuracy_1_of_2‚0.6486139297485352 particle/accuracy_4_of_5‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1309555023908615 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9750211834907532 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9764621257781982 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.22208388149738312 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7200000286102295 ,jet/accuracy_3_of_3‚0.029538992792367935 jet/accuracy_2_of_5‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.14714768528938293 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.013598017394542694 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.3450619578361511 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.0811813697218895 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 particle/accuracy_2_of_3‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.47829487919807434 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4537234604358673 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.010416666977107525 5 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.50056391954422 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 jet/accuracy_3_of_5‚0 _runtime‚ 651.564347728 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.2992420792579651 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.3422773480415344 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8363640904426575 9 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.042118657380342484 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375:¤Oc›‚(:Ø é±õÒùãì¿ disk.nvme0c0n1.in‚ 47.203125$ memory_percent‚99.19359683990479) proc.memory.percent‚17.53404140472412 cpu‚26.664201994471497 proc.cpu.threads‚17$ disk.nvme1n1.out‚616.48583984375 disk.md0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚201.5810546875 disk.rbd0.out‚0$ disk.nvme0c0n1.out‚ 3650.41796875( proc.memory.availableMB‚ 132.12109375# proc.memory.rssMB‚ 2872.77734375 network.sent‚ 1.913015e+06 network.recv‚233449 disk.nvme1n1.in‚ 5.98828125" disk.dm-0.out‚4023.94287109375) disk./.usagePercent‚75.88682089269676$ disk./.usageGB‚1165.052417755127 disk.nvme0n1.in‚ 47.203125" disk.nvme0n1.out‚ 3650.41796875 disk.nvme1c1n1.in‚ 5.98828125 disk.dm-0.in‚ 53.19140625 disk.rbd0.in‚0& disk.nvme1c1n1.out‚616.48583984375 disk.md0.out‚ 19.43359375Un¬‹›œ‚(: é±õÒß À gpu.0.gpu‚60.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚201.152' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.05066666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—™Ù0+ _wandbruntime‚669  _runtime‚669½IšjÇžÁ ' lr-AdamW/pg2‚0.00011199791882281668 trainer/global_step‚1649 % _timestamp‚1.7845025155458808e+09 _runtime‚ 669.60926842 _step‚130 ' lr-AdamW/pg1‚0.00011199791882281668ŒN@KÌŸÆ ' lr-AdamW/pg1‚0.00011199791882281668 ' lr-AdamW/pg2‚0.00011199791882281668 trainer/global_step‚1649 % _timestamp‚1.7845025155458808e+09 _runtime‚ 669.60926842 _step‚130‚²±P0 + _wandbruntime‚669  _runtime‚669È›¡¸O XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚63609234310144 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚63942674411795 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚62785154258416 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚63609251349504 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1161146843648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4246062 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60764366443520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚63942571223965 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29283000 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1157520153379 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚64770293132288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196929024 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1161058822144 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚63609234310144 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12558573 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172025344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1194895087616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16734569 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16724427 trainer/global_step‚1649 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11515134 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 _runtime‚ 669.609731362 _step‚131 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚61959089505792 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1161058822144 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚64770293132288 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29293144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11514906 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152ßkpTÃ'¢½O A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚63609251349504 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1161146843648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4246062 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60764366443520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚63942571223965 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29283000 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1157520153379 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚64770293132288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196929024 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1161058822144 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚63609234310144 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12558573 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172025344 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1194895087616 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚16734569 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚16724427 trainer/global_step‚1649 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚11515134 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 _runtime‚ 669.609731362 _step‚131 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚61959089505792 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1161058822144 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚64770293132288 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29293144 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11514906 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚63609234310144 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚63942674411795 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚62785154258416 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482ƒ:ë0£+ _wandbruntime‚669  _runtime‚669 z§È&¤ÂM >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1161736931840 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 $ _timestamp‚1.784502515797816e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1161824960512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1158196104935 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚64808337609216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚63980119243169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16734325 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1161736931840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4248725 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚60800086504448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚60800061600768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11521618 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406784 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚62822009286816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚64808442677248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1158109956353 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16744481 trainer/global_step‚1649 _runtime‚ 669.861197169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29310377 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12565896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12565894 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚62822026326176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚63980222431111 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7273137 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1195420385280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29310377 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚63646617716736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11521862 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4248484 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12565896 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482PÙ #|!¥ÇM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1161824960512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1158196104935 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚64808337609216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚63980119243169 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16734325 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1161736931840 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4248725 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚60800086504448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚60800061600768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11521618 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406784 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚62822009286816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚64808442677248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1158109956353 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16744481 trainer/global_step‚1649 _runtime‚ 669.861197169 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29310377 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12565896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12565894 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚62822026326176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚63980222431111 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7273137 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1195420385280 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29310377 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 REDeviceSt‚Æ5’QatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚63646617716736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11521862 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4248484 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12565896 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1161736931840 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 $ _timestamp‚1.784502515797816e+09„­(Öü0¦+ _wandbruntime‚669  _runtime‚669Bð3º§þ &loss/total_loss‚20.795679121826673 % _timestamp‚1.7845025157993822e+09 2loss/SRqqt2/assignment_loss‚1.1171974697409763 1loss/FBt1/assignment_loss‚0.09463109680707427 /loss/FRt2/detection_loss‚1.2870831445607165 /loss/FBt2/detection_loss‚3.7616522612341385 _runtime‚ 669.862239072 0loss/FRt1/assignment_loss‚3.6590428388208376 2loss/SRqqt1/assignment_loss‚1.5087236050320298 0loss/FBt2/assignment_loss‚0.2080161515409405 /loss/FRt1/detection_loss‚0.5932692376164985 epoch‚2 /loss/FRt2/assignment_loss‚1.991136405223645 1loss/SRqqt2/detection_loss‚0.6930828542731003 trainer/global_step‚1649 _step‚133 1loss/SRqqt1/detection_loss‚2.2717113367919075 .loss/FBt1/detection_loss‚3.610132720184811YÒð® ¨ƒ 1loss/SRqqt1/detection_loss‚2.2717113367919075 .loss/FBt1/detection_loss‚3.610132720184811 &loss/total_loss‚20.795679121826673 % _timestamp‚1.7845025157993822e+09 2loss/SRqqt2/assignment_loss‚1.1171974697409763 1loss/FBt1/assignment_loss‚0.09463109680707427 /loss/FRt2/detection_loss‚1.2870831445607165 /loss/FBt2/detection_loss‚3.7616522612341385 _runtime‚ 669.862239072 0loss/FRt1/assignment_loss‚3.6590428388208376 2loss/SRqqt1/assignment_loss‚1.5087236050320298 0loss/FBt2/assignment_loss‚0.2080161515409405 /loss/FRt1/detection_loss‚0.5932692376164985 epoch‚2 /loss/FRt2/assignment_loss‚1.991136405223645 1loss/SRqqt2/detection_loss‚0.6930828542731003 trainer/global_step‚1649 _step‚133…¯î†Z©‚(:Ï ø±õÒÚ×ì¿# proc.memory.rssMB‚ 2872.77734375 network.sent‚ 1.957077e+06 disk.nvme0c0n1.in‚47.21875! disk.nvme0n1.out‚ 3658.1015625$ disk.nvme1n1.out‚681.35107421875 disk.md0.in‚0) proc.memory.percent‚17.53404140472412 proc.cpu.threads‚17 network.recv‚237241% disk./.usageGB‚1165.3042068481445 disk.rbd0.in‚0 disk.nvme0n1.in‚47.21875& disk.nvme1c1n1.out‚681.35107421875 disk.dm-0.in‚ 53.2109375' proc.memory.availableMB‚ 132.1171875 cpu‚26.669335155365527 disk.rbd0.out‚0# disk.nvme0c0n1.out‚ 3658.1015625 disk.md0.out‚ 19.8828125) disk./.usagePercent‚75.90322141984332 disk.nvme1c1n1.in‚ 5.9921875 disk.nvme1n1.in‚ 5.9921875" disk.dm-0.out‚4031.20263671875 disk.dm-1.in‚0 disk.dm-1.out‚265.9208984375# memory_percent‚99.1936206817627ö˜#Ä›ª‚(: ø±õÒí™–À gpu.0.gpu‚21.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚111.695' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.23166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰Ž10«+ _wandbruntime‚688  _runtime‚688•.þȬ % _timestamp‚1.7845025343047576e+09 _runtime‚ 688.369408872 _step‚134 ' lr-AdamW/pg1‚0.00011198225299700319 ' lr-AdamW/pg2‚0.00011198225299700319 trainer/global_step‚1699ÒÍ"Í­Ç % _timestamp‚1.7845025343047576e+09 _runtime‚ 688.369408872 _step‚134 ' lr-AdamW/pg1‚0.00011198225299700319 ' lr-AdamW/pg2‚0.00011198225299700319 trainer/global_step‚1699† ©b&0®+ _wandbruntime‚688  _runtime‚688Ƴ¾'¯¸O E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚64628757646416 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 $ _timestamp‚1.784502534307073e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚64628740607056 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚135 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17221179 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1194996563968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184608256 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12924625 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1195084585472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17231321 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17231321 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4368752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚66672549235712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17231321 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17231321 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4368752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚66672549235712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚135 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17221179 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1194996563968 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184608256 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12924625 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1195084585472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17231077 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11858788 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62584844366848 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1191939431761 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚63815229455872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚1699 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30173179 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30173179 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62584819463168 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚65514919038976 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚64665629714176 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12931944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12931944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11859035 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7845025345581117e+09 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30163021 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1195762702336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚64665612674816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚65857552106577 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚65514936078336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30163021 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213699072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1195762702336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188795392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚66710698780672 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 _runtime‚ 688.621431177 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17241233 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚30163021 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1195762702336 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚64665612674816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚65857552106577 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚65514936078336 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30163021 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213699072 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1195762702336 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188795392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚66710698780672 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 _runtime‚ 688.621431177 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17241233 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17231077 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11858788 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚62584844366848 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1191939431761 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚63815229455872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚1699 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚30173179 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30173179 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚62584819463168 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚65514919038976 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚64665629714176 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12931944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚12931944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚11859035 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7845025345581117e+09 BµW·‚(:Ì ‡²õÒÝâ‘À network.sent‚ 2.031606e+06) disk./.usagePercent‚75.90322117136931 disk.nvme1n1.in‚ 5.9921875 disk.md0.out‚ 20.28515625 disk.dm-0.out‚4041.318359375& disk.nvme0c0n1.out‚3668.3896484375" disk.nvme1c1n1.out‚ 682.1328125# memory_percent‚99.5891809463501 network.recv‚241191 disk.nvme1n1.out‚ 682.1328125$ disk.nvme0n1.out‚3668.3896484375' proc.memory.availableMB‚ 67.30859375# proc.memory.rssMB‚ 2872.77734375% disk./.usageGB‚1165.3042030334473 disk.md0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚47.21875) proc.memory.percent‚17.53404140472412 cpu‚26.674226326563375 proc.cpu.threads‚17 disk.dm-0.in‚ 53.2109375 disk.dm-1.in‚0 disk.dm-1.out‚266.0224609375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚47.21875 disk.nvme1c1n1.in‚ 5.9921875Ís ¸‚(:… ‡²õÒÏß»À gpu.0.gpu‚63.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚202.437' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚67.479 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôÙab¹‚(:× –²õÒÊ´”À! disk.dm-1.out‚266.30615234375 disk.rbd0.out‚0& disk.nvme0c0n1.out‚5661.7822265625$ disk.nvme1n1.out‚682.98974609375 disk.dm-0.in‚ 53.2109375' proc.memory.availableMB‚ 67.25390625 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd0.in‚0$ disk.nvme0n1.out‚5661.7822265625) proc.memory.percent‚17.53404140472412 network.recv‚245293% disk./.usageGB‚1167.6468696594238 disk.nvme0c0n1.in‚47.21875 disk.md0.in‚0 disk.md0.out‚ 20.69140625$ memory_percent‚99.58951473236084 cpu‚26.659464701378898 network.sent‚ 2.082897e+06 disk.nvme0n1.in‚47.21875 disk.nvme1c1n1.in‚ 5.9921875& disk.nvme1c1n1.out‚682.98974609375 disk.nvme1n1.in‚ 5.9921875! disk.dm-0.out‚6034.4111328125# proc.memory.rssMB‚ 2872.77734375) disk./.usagePercent‚76.05581303757849 Q¼º‚(:… –²õÒ¤³¾À gpu.0.gpu‚63.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.933' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.311 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®t¨0»+ _wandbruntime‚706  _runtime‚706lÛ"ǼÁ _runtime‚ 706.957163731 _step‚138 ' lr-AdamW/pg1‚0.00011196569151899014 ' lr-AdamW/pg2‚0.00011196569151899014 trainer/global_step‚1749 $ _timestamp‚1.784502552893433e+09øX Ì½Æ trainer/global_step‚1749 $ _timestamp‚1.784502552893433e+09 _runtime‚ 706.957163731 _step‚138 ' lr-AdamW/pg1‚0.00011196569151899014 ' lr-AdamW/pg2‚0.00011196569151899014Š·Ž˜¿0¾+ _wandbruntime‚706  _runtime‚706ª¾æ¿'¿¹O O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚31018748 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4492612 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31008604 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚67345871033344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1228934305792 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12190456 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7697847 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1225092955725 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68574805339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7697844 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65598531922432 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚65598737240064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845025528941076e+09 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1264649754112 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚67345888072704 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1229022327296 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64333882168320 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13290673 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚66472361034416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚67345888072704 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13290675 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17728073 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚67697436950781 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚67345871033344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13290675 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12190707 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚66472343995056 trainer/global_step‚1749 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 _step‚139 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚68574910400000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1229022327296 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚68574805339136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1228934305792 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17717931 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚67345888072704 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1229022327296 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64333882168320 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13290673 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚66472361034416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚67345888072704 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13290675 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17728073 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚67697436950781 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚67345871033344 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13290675 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12190707 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚66472343995056 trainer/global_step‚1749 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 _step‚139 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚68574910400000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1229022327296 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚68574805339136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1228934305792 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17717931 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1225092955725 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68574805339136 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7697844 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚65598531922432 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482‹±À0Á+ _wandbruntime‚707  _runtime‚707¶>cÈ&ÂÂM E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1229700444160 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7702134 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213699072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚66509216062816 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1225768907169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚67383254439936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1225855055751 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17737985 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1229612415488 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1229700444160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚67734984969985 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚ 707.216732902 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚67383254439936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31035981 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13297996 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1265343702016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68612849816064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12197232 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚68612954884096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1265532497408 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7702137 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13297994 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64369577325568 trainer/global_step‚1749 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚66509233102176 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13297994 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 $ _timestamp‚1.784502553153417e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4495337 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 _step‚140 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31025823 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚67383237400576 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31025823 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64369602229248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188795392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0ÎÍÍ&ÃÇM E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 $ _timestamp‚1.784502553153417e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4495337 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 _step‚140 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31025823 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚67383237400576 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31025823 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64369602229248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188795392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1229700444160 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7702134 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213699072 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚66509216062816 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1225768907169 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚67383254439936 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1225855055751 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17737985 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1229612415488 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1229700444160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚67734984969985 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _runtime‚ 707.216732902 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚67383254439936 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31035981 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13297996 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1265343702016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68612849816064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12197232 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚68612954884096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1265532497408 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7702137 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13297994 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64369577325568 trainer/global_step‚1749 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚66509233102176 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13297994Œr ]×0Ä+ _wandbruntime‚707  _runtime‚707ÖUGÆÅú 0loss/FRt2/assignment_loss‚2.2025247550010683 1loss/SRqqt2/detection_loss‚0.6853318956480718 .loss/FBt1/detection_loss‚2.965163507818968 trainer/global_step‚1749 % _timestamp‚1.7845025531543643e+09 /loss/FRt1/assignment_loss‚3.578856190533837 /loss/FBt2/assignment_loss‚0.209843062371235 .loss/FRt2/detection_loss‚1.289811104969435 epoch‚2 _runtime‚ 707.217179022 1loss/FBt1/assignment_loss‚0.29860157680069505 /loss/FRt1/detection_loss‚0.5527833634933127 2loss/SRqqt1/assignment_loss‚1.5197911119331484 2loss/SRqqt2/assignment_loss‚1.1592890355197707 0loss/SRqqt1/detection_loss‚2.110283449267824 .loss/FBt2/detection_loss‚2.267090333227659 &loss/total_loss‚18.839369386585027 _step‚141o"Æÿ _step‚141 0loss/FRt2/assignment_loss‚2.2025247550010683 1loss/SRqqt2/detection_loss‚0.6853318956480718 .loss/FBt1/detection_loss‚2.965163507818968 trainer/global_step‚1749 % _timestamp‚1.7845025531543643e+09 /loss/FRt1/assignment_loss‚3.578856190533837 /loss/FBt2/assignment_loss‚0.209843062371235 .loss/FRt2/detection_loss‚1.289811104969435 epoch‚2 _runtime‚ 707.217179022 1loss/FBt1/assignment_loss‚0.29860157680069505 /loss/FRt1/detection_loss‚0.5527833634933127 2loss/SRqqt1/assignment_loss‚1.5197911119331484 2loss/SRqqt2/assignment_loss‚1.1592890355197707 0loss/SRqqt1/detection_loss‚2.110283449267824 .loss/FBt2/detection_loss‚2.267090333227659 &loss/total_loss‚18.839369386585027PG6=aÇ‚(:Ö ¥²õÒ“ÄÞ¿ network.recv‚248895" disk.dm-0.out‚6232.26806640625 disk.rbd0.in‚0 disk.nvme1c1n1.in‚ 5.9921875& disk.nvme1c1n1.out‚745.45263671875) proc.memory.percent‚17.53404140472412% disk./.usageGB‚1168.3036651611328 disk.nvme1n1.in‚ 5.9921875$ disk.nvme1n1.out‚745.45263671875 disk.md0.out‚ 21.25390625 disk.dm-0.in‚ 53.2109375! disk.dm-1.out‚266.39990234375 disk.nvme0n1.in‚47.21875$ proc.memory.availableMB‚67.15625 network.sent‚ 2.086021e+06 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚47.21875' disk.nvme0c0n1.out‚5798.46142578125% disk.nvme0n1.out‚5798.46142578125# proc.memory.rssMB‚ 2872.77734375 cpu‚26.644658822505676) disk./.usagePercent‚76.09859405055417 disk.md0.in‚0# memory_percent‚99.5901107788086 proc.cpu.threads‚17ãÑ6w›È‚(: ¥²õÒ…ÿ´À gpu.0.gpu‚56.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚192.782' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚64.26066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»$Ù_0É+ _wandbruntime‚725  _runtime‚7251óë•ÈÊ trainer/global_step‚1799 % _timestamp‚1.7845025718288014e+09 _runtime‚ 725.893904222 _step‚142 ' lr-AdamW/pg1‚0.00011194823465404322 ' lr-AdamW/pg2‚0.00011194823465404322­¾â·ÍËÇ ' lr-AdamW/pg2‚0.00011194823465404322 trainer/global_step‚1799 % _timestamp‚1.7845025718288014e+09 _runtime‚ 725.893904222 _step‚142 ' lr-AdamW/pg1‚0.00011194823465404322Ž 5H0Ì+ _wandbruntime‚725  _runtime‚725“Í$•˜Í¹O TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13656725 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316800 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚69574869814189 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1262960069120 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4615268 _step‚143 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18214683 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1259008579603 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31871406 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12527854 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1258922431133 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13656725 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18224825 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1262872047616 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚66118664934400 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitòq ·'or.on_train_batch_start/inactive_split.all.freed‚12527612 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚69214206434304 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1262960069120 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7912344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚69214206434304 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚70477166503424 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4615507 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203220480 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚68315947383056 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1299576479744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67418419730944 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚69574973002019 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70477061442560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚66118640030720 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330¢Ž£VÄ'ξO XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚70477166503424 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4615507 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203220480 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚68315947383056 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1299576479744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67418419730944 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚69574973002019 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70477061442560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚66118640030720 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13656725 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316800 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚69574869814189 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1262960069120 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4615268 _step‚143 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18214683 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1259008579603 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚31871406 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12527854 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1258922431133 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13656725 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18224825 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1262872047616 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚66118664934400 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12527612 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚69214206434304 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1262960069120 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7912344 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚69214206434304 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330·ÎáÉ0Ï+ _wandbruntime‚726  _runtime‚726m˜ÇNÉ&ÐÃM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7916637 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚69251572801536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1263550157312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4617741 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18224581 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13664044 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13664046 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217893376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13664044 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31898783 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18224581 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67454604037632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1300461839360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1263638185984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1259598382577 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚69251572801536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1300268849664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31898783 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚69251555762176 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31888625 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12534375 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67454821931008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚69612417833393 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7845025720943205e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚68352836490176 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31888625 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1263638185984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18234737 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13664046 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18234737 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12534631 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4617994 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70515210987520 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚69251555762176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448b ŒïùÑÈM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1263638185984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1259598382577 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚69251572801536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1300268849664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚31898783 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚144 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚69251555762176 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚31888625 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12534375 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚67454821931008 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚69612417833393 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7845025720943205e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚68352836490176 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚31888625 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1263638185984 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18234737 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13664046 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18234737 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12534631 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4617994 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70515210987520 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚69251555762176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7916637 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚69251572801536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1263550157312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4617741 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18224581 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13664044 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 M>Devic\+ôÕ eStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13664046 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217893376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13664044 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚31898783 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18224581 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚67454604037632 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1300461839360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0¸0Ò+ _wandbruntime‚726  _runtime‚726uƒöÿÓù /loss/FRt1/detection_loss‚0.5140698099738873 1loss/SRqqt1/detection_loss‚2.6403281891947255 1loss/SRqqt2/detection_loss‚0.7063713900635321 # _timestamp‚1.78450257209536e+09 0loss/FRt2/assignment_loss‚2.2851322875692954 0loss/FBt2/assignment_loss‚0.1408515747469358 _runtime‚ 726.158202304 _step‚145 2loss/SRqqt2/assignment_loss‚1.2498489690813244 /loss/FRt2/detection_loss‚1.3994089224445048 2loss/SRqqt1/assignment_loss‚1.4733941788344007 +loss/FBt1/detection_loss‚4.373848478923 /loss/FBt2/detection_loss‚1.3295444796290234 %loss/total_loss‚19.79713897269627 epoch‚2 trainer/global_step‚1799 0loss/FRt1/assignment_loss‚3.5902214335232245 1loss/FBt1/assignment_loss‚0.09411925871241643Q:Ôþ 2loss/SRqqt1/assignment_loss‚1.4733941788344007 +loss/FBt1/detection_loss‚4.373848478923 /loss/FBt2/detection_loss‚1.3295444796290234 %loss/total_loss‚19.79713897269627 epoch‚2 trainer/global_step‚1799 0loss/FRt1/assignment_loss‚3.5902214335232245 1loss/FBt1/assignment_loss‚0.09411925871241643 /loss/FRt1/detection_loss‚0.5140698099738873 1loss/SRqqt1/detection_loss‚2.6403281891947255 1loss/SRqqt2/detection_loss‚0.7063713900635321 # _timestamp‚1.78450257209536e+09 0loss/FRt2/assignment_loss‚2.2851322875692954 0loss/FBt2/assignment_loss‚0.1408515747469358 _runtime‚ 726.158202304 _step‚145 2loss/SRqqt2/assignment_loss‚1.2498489690813244 /loss/FRt2/detection_loss‚1.3994089224445048‘/©bÕ‚(:× ´²õҦDŽÀ$ memory_percent‚99.59070682525635# proc.memory.rssMB‚ 2872.77734375 proc.cpu.threads‚17 disk.nvme1c1n1.in‚ 5.9921875% disk.nvme1c1n1.out‚750.2373046875# disk.nvme1n1.out‚750.2373046875 disk.dm-0.in‚ 53.2109375" disk.dm-0.out‚6232.81103515625 disk.rbd0.out‚0 network.sent‚ 2.161145e+06 network.recv‚253283% disk./.usageGB‚1168.3038177490234 disk.nvme0c0n1.in‚47.21875 disk.nvme0n1.in‚47.21875 disk.md0.out‚ 21.67578125 disk.dm-1.in‚0' proc.memory.availableMB‚ 67.05859375) disk./.usagePercent‚76.09860398951463 disk.nvme1n1.in‚ 5.9921875 disk.md0.in‚0 disk.dm-1.out‚270.6904296875 disk.rbd0.in‚0) proc.memory.percent‚17.53404140472412 cpu‚26.630392189351188' disk.nvme0c0n1.out‚5799.42041015625% disk.nvme0n1.out‚5799.42041015625á¯Ö‚(:„ ´²õÒÿ‹¯À gpu.0.gpu‚90.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚197.49' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚65.83 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†m-]0×+ _wandbruntime‚744  _runtime‚744Þ#7ªÈØ ' lr-AdamW/pg1‚0.00011192988268176953 ' lr-AdamW/pg2‚0.00011192988268176953 trainer/global_step‚1849 % _timestamp‚1.7845025904685545e+09 _runtime‚ 744.532145399 _step‚146 Ôî;ÍÙÇ ' lr-AdamW/pg1‚0.00011192988268176953 ' lr-AdamW/pg2‚0.00011192988268176953 trainer/global_step‚1849 % _timestamp‚1.7845025904685545e+09 _runtime‚ 744.532145399 _step‚146’œë·0Ú+ _wandbruntime‚744  _runtime‚744¾(ð¿'Û¹O \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14022773 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚70159550771056 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14022775 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚71082524795904 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 % _timestamp‚1.7845025904692593e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚72379317545984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚71082507756544 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12865148 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18721577 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18711435 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4738301 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67903397893120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12864893 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚71082524795904 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1296809789440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1296897810944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32744352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14022775 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4738049 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚71452405865427 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32734208 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚71452302677597 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18721577 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚69237920691200 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67903422796800 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1292751906541 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 trainer/global_step‚1849 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1296897810944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18711435 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚72379317545984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _runtime‚ 744.532358319 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8126847 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1292838055011 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8126844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330kÄ'ܾO \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14022773 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚70159550771056 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14022775 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚71082524795904 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 % _timestamp‚1.7845025904692593e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚72379317545984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚71082507756544 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12865148 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18721577 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚18711435 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4738301 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67903397893120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12864893 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚71082524795904 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1296809789440 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1296897810944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚32744352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14022775 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4738049 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚71452405865427 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32734208 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚71452302677597 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚18721577 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚69237920691200 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67903422796800 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1292751906541 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 trainer/global_step‚1849 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1296897810944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚18711435 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚72379317545984 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _runtime‚ 744.532358319 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8126847 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1292838055011 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8126844 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330“æ,õr0Ý+ _wandbruntime‚744  _runtime‚744”²¦ýÞÃM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚71119874123776 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8131134 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚71119874123776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 % _timestamp‚1.7845025907252924e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1297575927808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14030094 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67939117954048 _step‚148 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8131137 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1335405254656 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚70196439878176 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14030096 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18721333 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1297487899136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32761585 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205310464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚71119891163136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67939093050368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚70196422838816 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14030096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚71489953884743 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32751427 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚72417467090944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚72417362022912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14030094 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚69274317898240 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚71489850696801 _runtime‚ 744.788625787 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32751427 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18721333 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12871687 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1293427857985 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1297487899136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18731489äøc=Î&ßÈM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32761585 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12871957 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚18731489 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1335224847872 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14030096 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚71489953884743 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚32751427 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚72417467090944 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚72417362022912 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14030094 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚69274317898240 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚71489850696801 _runtime‚ 744.788625787 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32751427 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18721333 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12871687 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1293427857985 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1297487899136 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18731489 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚71119874123776 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8131134 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚71119874123776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 % _timestamp‚1.7845025907252924e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1297575927808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14030094 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚67939117954048 _step‚148 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8131137 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1335405254656 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚70196439878176 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14030096 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚18721333 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1297487899136 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32761585 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205310464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚71119891163136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67939093050368 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚70196422838816 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414784 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚69688180659200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 _step‚151 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚72003171198416 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚69688155755520 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1369721980416 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14388825 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71057695224832 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13203115 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74281573649408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1326581381949 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14388823 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71057902639616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 _runtime‚ 763.326121443 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1330747531264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚73329735541005 % _timestamp‚1.7845026092616556e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1330747531264 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19208187 trainer/global_step‚1899 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8341344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19218329 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚72003154159056 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14388825 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8341347 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚72950826118144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚72950843157504 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72950826118144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4861771 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74281678710272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1330835552768 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1330835552768|>ID2ê¾O GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1330747531264 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19208187 trainer/global_step‚1899 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8341344 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚19218329 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚72003154159056 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚14388825 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8341347 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚72950826118144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚72950843157504 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚72950826118144 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4861771 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74281678710272 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1330835552768 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1330835552768 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4862009 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚74281573649408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚72950843157504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚73329838728835 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1326667530419 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1369539469312 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚19208187 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414784 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚69688180659200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 O@DeviceStat×{2’sMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 _step‚151 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚72003171198416 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚69688155755520 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1369721980416 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚14388825 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71057695224832 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13203115 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74281573649408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1326581381949 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14388823 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71057902639616 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 _runtime‚ 763.326121443 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182511104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1330747531264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚73329735541005 % _timestamp‚1.7845026092616556e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373— à^‘0ë+ _wandbruntime‚763  _runtime‚763-k­<É&ìÃM BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚73367283560209 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1327257333393 _step‚152 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7845026095176883e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33614229 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14396146 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚74319618126336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚74319723194368 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71094084248064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚1899 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1331513669632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14396146 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19228241 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1331425640960 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14396144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚69723875816448 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14396144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1331425640960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚74319723194368 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19228241 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19218085 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13209835 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71094295849984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72988192485376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8345634 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33624387 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186698240 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚73367386748151 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72988209524736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13210076 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8345637 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4864439 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33624387 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚73367283560209 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1327257333393 _step‚152 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7845026095176883e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚33614229 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14396146 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚74319618126336 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚74319723194368 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71094084248064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚1899 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1331513669632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14396146 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19228241 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1331425640960 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14396144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚69723875816448 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14396144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1331425640960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚74319723194368 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19228241 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19218085 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13209835 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71094295849984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚72988192485376 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8345634 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚33624387 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186698240 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚73367386748151 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚72988209524736 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13210076 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8345637 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4864439 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚33624387 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76183829752832 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚72877582017024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚74819144479744 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14754873 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4985910 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚73846774586416 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1364773294592 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚76183829752832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚72877370407936 _runtime‚ 782.13965176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚74819144479744 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚73846757547056 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚121Ý0¯302 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76183934813696 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 trainer/global_step‚1949 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34469956 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19704939 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13542006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1364685273088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13541754 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34459812 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1404456790016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚74819161519104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚155 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚75207271592243 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845026280750628e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1364773294592 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8555844 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1364685273088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1404643495424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76183934813696 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1360410857357 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14754873 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19715081 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1360497005827 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4986159 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186705408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1404643495424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76183934813696 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1360410857357 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚14754873 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19715081 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1360497005827 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4986159 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186705408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76183829752832 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚72877582017024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚74819144479744 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚14754873 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4985910 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚73846774586416 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1364773294592 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚76183829752832 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚72877370407936 _runtime‚ 782.13965176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚74819144479744 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚73846757547056 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚76183934813696 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 trainer/global_step‚1949 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34469956 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚19704939 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13542006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1364685273088 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13541754 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚34459812 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1404456790016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚74819161519104 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚155 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚75207271592243 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0›Ÿå—0û+ _wandbruntime‚782  _runtime‚782. mëÉ&üÃM QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8560134 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14762194 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1405153494528 % _timestamp‚1.7845026283299458e+09 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34487189 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76221979297792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14762196 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1365451411456 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1405340192768 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚73883646654176 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 _step‚156 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚74856510846976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14762196 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19724993 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1361172957383 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 trainer/global_step‚1949 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76221874229760 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 _runtime‚ 782.393258798 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19724993 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14762194 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13548471 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13548719 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34477031 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚74856510846976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34477031 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚72913973871616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚73883629614816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚75244716423617 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚71508633678848 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34487189 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1365363382784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34487189 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1365363382784 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601920 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34487189 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚76221979297792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14762196 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1365451411456 OGDeviceStatsMonitor.on_train_batch_end/inactive_s-X¯í\plit.small_pool.current‚245 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1405340192768 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚73883646654176 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 _step‚156 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚74856510846976 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚14762196 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19724993 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1361172957383 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 trainer/global_step‚1949 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76221874229760 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 _runtime‚ 782.393258798 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19724993 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14762194 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13548471 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚13548719 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34477031 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚74856510846976 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34477031 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚72913973871616 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚73883629614816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚75244716423617œ¢¿ €0þ+ _wandbruntime‚782  _runtime‚782>~Öàÿü _step‚157 0loss/FRt2/assignment_loss‚2.0063413323554316 1loss/FBt2/assignment_loss‚0.09370274639146445 0loss/SRqqt1/detection_loss‚2.039541730101694 1loss/SRqqt2/detection_loss‚0.7625772833319643 1loss/FBt1/assignment_loss‚0.10173653265696037 /loss/FBt2/detection_loss‚1.5361701479656813 trainer/global_step‚1949 1loss/SRqqt1/assignment_loss‚1.263138775370623 /loss/FRt1/detection_loss‚0.6038846215246639 /loss/FRt2/detection_loss‚1.3318102174012023 -loss/FBt1/detection_loss‚2.23484810648883 %loss/total_loss‚16.26573296577631 epoch‚2 _runtime‚ 782.393721279 0loss/FRt1/assignment_loss‚3.2669109501899816 2loss/SRqqt2/assignment_loss‚1.0250705219978136 % _timestamp‚1.7845026283309033e+09ðh€  0loss/FRt2/assignment_loss‚2.0063413323554316 1loss/FBt2/assignment_loss‚0.09370274639146445 0loss/SRqqt1/detection_loss‚2.039541730101694 1loss/SRqqt2/detection_loss‚0.7625772833319643 1loss/FBt1/assignment_loss‚0.10173653265696037 /loss/FBt2/detection_loss‚1.5361701479656813 trainer/global_step‚1949 1loss/SRqqt1/assignment_loss‚1.263138775370623 /loss/FRt1/detection_loss‚0.6038846215246639 /loss/FRt2/detection_loss‚1.3318102174012023 -loss/FBt1/detection_loss‚2.23484810648883 %loss/total_loss‚16.26573296577631 epoch‚2 _runtime‚ 782.393721279 0loss/FRt1/assignment_loss‚3.2669109501899816 2loss/SRqqt2/assignment_loss‚1.0250705219978136 % _timestamp‚1.7845026283309033e+09 _step‚157&ÓfÁn ‚(:ã ð²õÒã´Ý¿ network.recv‚268713 disk.rbd0.in‚0' disk.nvme0c0n1.out‚6604.46533203125' disk.nvme1c1n1.out‚1018.16455078125 disk.nvme1n1.in‚ 10.48046875 disk.md0.out‚ 23.49609375! disk.dm-1.out‚271.25634765625& proc.memory.availableMB‚ 66.8359375% disk./.usageGB‚1168.2585105895996 disk.rbd0.out‚0! disk.nvme1c1n1.in‚ 10.48046875% disk.nvme1n1.out‚1018.16455078125 disk.dm-1.in‚0) proc.memory.percent‚17.53404140472412 cpu‚26.579028239330874( disk./.usagePercent‚76.0956528636769 disk.nvme0n1.in‚ 48.3671875% disk.nvme0n1.out‚6604.46533203125 disk.dm-0.in‚ 58.84765625" disk.dm-0.out‚7301.33837890625# proc.memory.rssMB‚ 2872.77734375 proc.cpu.threads‚17 network.sent‚ 2.339946e+06 disk.nvme0c0n1.in‚ 48.3671875 disk.md0.in‚0$ memory_percent‚99.59206581115723º;ˆ!œ‚ ‚(:‘ ð²õÒžú¿À gpu.0.gpu‚84.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚119.563' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.85433333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úOêÚ0ƒ + _wandbruntime‚801  _runtime‚8013_‹‡È„  ' lr-AdamW/pg1‚0.00011186945913208518 ' lr-AdamW/pg2‚0.00011186945913208518 trainer/global_step‚1999 % _timestamp‚1.7845026469663572e+09 _runtime‚ 801.029502191 _step‚158Õ›teÍ… Ç _runtime‚ 801.029502191 _step‚158 ' lr-AdamW/pg1‚0.00011186945913208518 ' lr-AdamW/pg2‚0.00011186945913208518 trainer/global_step‚1999 % _timestamp‚1.7845026469663572e+09žJoÍ0† + _wandbruntime‚801  _runtime‚801i£‚¿'‡ ¹O UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚78086190917120 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73257696384000 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15120925 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186705408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 % _timestamp‚1.7845026469672015e+09 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1398711036416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1398623014912 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8770344 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35332758 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1394240332765 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1398711036416 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15120925 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20201691 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5109626 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚75690377974416 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211609088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1394326481235 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5109868 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 _step‚159 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15120923 _runtime‚ 801.033566569 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚76687479880704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78086085856256 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚73257671480320 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1439390562816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1398623014912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1439577268224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚78086190917120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74697273652224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚77084601267821 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚78086085856256 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚77084704455651 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚76687462841344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20211833 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8770347 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13880215 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚75690360935056 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35322614 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20211833 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚76687462841344 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15120923 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚74697062043136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20201691 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13879970 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 trainer/global_step‚1999 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78086085856256 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚73257671480320 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1439390562816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1398623014912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1439577268224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚78086190917120 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74697273652224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚77084601267821 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚78086085856256 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚77084704455651 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚76687462841344 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20211833 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8770347 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13880215 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚75690360935056 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35322614 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20211833 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚76687462841344 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15120923 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚74697062043136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20201691 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13879970 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 trainer/global_step‚1999 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚75727250042176 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚77122149287025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35339833 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15128244 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚76724829208576 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15128244 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8774637 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35339833 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/segmenèbéCrt.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚78124130333184 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1394916284209 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35349991 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20221745 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74733452053504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚ 801.289061401 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15128246 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5112050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚76724829208576 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75727233002816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77122252474967 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7845026472257264e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20211589 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13886684 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1440085415936 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35349991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76724846247936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1440278405632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989696 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152¿-ÆëÎ&‹ ÈM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35339833 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚78124130333184 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1394916284209 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35349991 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20221745 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74733452053504 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚ 801.289061401 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15128246 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5112050 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚76724829208576 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚75727233002816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77122252474967 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7845026472257264e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20211589 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13886684 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1440085415936 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35349991 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76724846247936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1440278405632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989696 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1399301124608 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8774634 trainer/global_step‚1999 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚73293366637568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚75727250042176 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚77122149287025 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35339833 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15128244 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚76724829208576 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15128244 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8774637 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79988341959680 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36195560 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15486975 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚79988341959680 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚76516760284672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8984847 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14218117 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚78962137319059 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15486973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2049 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1432648778240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20708585 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚79988447020544 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5233517 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1432560756736 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚79988447020544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36185416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15486973 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1474511355904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚163 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14218364 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚79988447020544 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚36185416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15486973 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1474511355904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚163 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14218364 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79988341959680 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚36195560 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15486975 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚79988341959680 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚76516760284672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8984847 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14218117 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚78962137319059 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15486973 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚2049 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1432648778240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20708585 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚79988447020544 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5233517 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1432560756736 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36212793 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36202635 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76553155628032 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚80026491504640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1433326895104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚80026491504640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚77570853430176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚77570836390816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14224875 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15494294 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1433326895104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1433238866432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8989137 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1428831908199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5235741 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚75078149403648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20718497 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20708341 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20708341 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚78593164609536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15494294 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5235970 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36202635 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚75078124499968 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176212480 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚78999582150433 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14225107 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚229 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚80026386436608 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1433238866432 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8989134 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15494296 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76553356744192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201116160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚78593147570176 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20718497 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚78593147570176 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816³p†uÎ&™ ÈM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚36212793 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚36212793 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚36202635 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76553155628032 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚80026491504640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1433326895104 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚80026491504640 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚77570853430176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚77570836390816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14224875 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15494294 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1433326895104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1433238866432 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8989137 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1428831908199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5235741 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚75078149403648 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20718497 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20708341 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20708341 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚78593164609536 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15494294 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5235970 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚36202635 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚75078124499968 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176212480 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚78999582150433 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14225107 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚229 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚80026386436608 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1433238866432 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8989134 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15494296 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚76553356744192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201116160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚78593147570176 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20718497 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚78593147570176 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _step‚164¤%š0š + _wandbruntime‚819  _runtime‚819«ä› € 0loss/FRt2/assignment_loss‚2.0375588916963148 2loss/SRqqt1/assignment_loss‚1.3112780113789169 /loss/FRt1/detection_loss‚0.5840498549471715 &loss/total_loss‚17.472302722920713 % _timestamp‚1.7845026658386266e+09 /loss/FBt1/detection_loss‚1.9573462349637663 _runtime‚ 819.901641638 2loss/SRqqt2/assignment_loss‚1.2745094876963554 1loss/FBt1/assignment_loss‚0.16177985401771844 1loss/FBt2/assignment_loss‚0.10805732084372721 1loss/SRqqt2/detection_loss‚0.7118191637901717 .loss/FBt2/detection_loss‚2.677015638389803 /loss/FRt2/detection_loss‚1.3689503463652668 1loss/SRqqt1/detection_loss‚1.8311610604655117 epoch‚2 trainer/global_step‚2049 _step‚165 0loss/FRt1/assignment_loss‚3.4487768583659886S¥¨ œ … .loss/FBt2/detection_loss‚2.677015638389803 /loss/FRt2/detection_loss‚1.3689503463652668 1loss/SRqqt1/detection_loss‚1.8311610604655117 epoch‚2 trainer/global_step‚2049 _step‚165 0loss/FRt1/assignment_loss‚3.4487768583659886 0loss/FRt2/assignment_loss‚2.0375588916963148 2loss/SRqqt1/assignment_loss‚1.3112780113789169 /loss/FRt1/detection_loss‚0.5840498549471715 &loss/total_loss‚17.472302722920713 % _timestamp‚1.7845026658386266e+09 /loss/FBt1/detection_loss‚1.9573462349637663 _runtime‚ 819.901641638 2loss/SRqqt2/assignment_loss‚1.2745094876963554 1loss/FBt1/assignment_loss‚0.16177985401771844 1loss/FBt2/assignment_loss‚0.10805732084372721 1loss/SRqqt2/detection_loss‚0.7118191637901717¥jM‘5i ‚(:Þ Ž³õÒ¨µ†À network.sent‚ 2.477741e+06 network.recv‚277844' disk.nvme0c0n1.out‚6676.54833984375! disk.nvme1c1n1.in‚ 10.48046875# proc.memory.rssMB‚ 2872.77734375) proc.memory.percent‚17.53404140472412 proc.cpu.threads‚17% disk./.usageGB‚1168.2585144042969 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 48.3671875% disk.nvme0n1.out‚6676.54833984375 disk.nvme1n1.in‚ 10.48046875 disk.md0.out‚ 24.15234375 disk.dm-0.out‚ 7424.27734375! disk.dm-1.out‚271.51904296875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 48.3671875& disk.nvme1c1n1.out‚1070.7041015625$ disk.nvme1n1.out‚1070.7041015625 disk.md0.in‚0 disk.dm-0.in‚ 58.84765625& proc.memory.availableMB‚ 66.7109375) disk./.usagePercent‚76.09565311215091$ memory_percent‚99.59282875061035 cpu‚26.55562270555006@Á¼ž ‚(:„ Ž³õÒÁ£¸À gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚194.838' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚64.946 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à¡k[0Ÿ + _wandbruntime‚838  _runtime‚838È‚äÏÈ   ' lr-AdamW/pg2‚0.00011182470700006642 trainer/global_step‚2099 % _timestamp‚1.7845026845334134e+09 _runtime‚ 838.598565879 _step‚166 ' lr-AdamW/pg1‚0.000111824707000066423/ö Í¡ Ç ' lr-AdamW/pg2‚0.00011182470700006642 trainer/global_step‚2099 % _timestamp‚1.7845026845334134e+09 _runtime‚ 838.598565879 _step‚166 ' lr-AdamW/pg1‚0.00011182470700006642¦Å-ÃÒ0¢ + _wandbruntime‚838  _runtime‚838ÙüÂX £ ¹O O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14556248 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚78336668427264 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚76827212108800 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚79377584750416 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚80424099564544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211609088 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9199344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1509456318464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMáN$½gonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1466586520064 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚78336456818176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15853025 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚81890703123968 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚80424099564544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76827187205120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚167 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21195195 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21195195 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1466498498560 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14555994 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5356901 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1461985432051 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚2099 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1466498498560 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37058362 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1461899283581 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚81890598063104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15853023 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚80424116603904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚80424116603904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21205337 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80839466994637 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15853025 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21205337 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1466586520064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9199347 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚81890598063104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15853023 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚80424116603904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚80424116603904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21205337 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80839466994637 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚15853025 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21205337 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1466586520064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9199347 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚80424099564544 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211609088 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9199344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1509456318464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1466586520064 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚78336456818176 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15853025 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚81890703123968 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚80424099564544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76827187205120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚167 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21195195 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21195195 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1466498498560 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14555994 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5356901 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1461985432051 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚2099 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1466498498560 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37058362 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1461899283581 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1467176608256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15860344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14562746 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1462575235025 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚81928747608064 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚80461465931776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚80461482971136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1510151385600 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21205093 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5359112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9203634 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199019008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1462661383607 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 _runtime‚ 838.854492232 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37075595 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21215249 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14563007 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚76862882362368 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21205093 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚80461465931776 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚78372859632640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚2099 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15860344 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37065437 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21215249 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37075595 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1467264636928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 % _timestamp‚1.7845026847911966e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5359370 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚81928747608064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37065437 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15860346 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚76862907266048WЯɹ§ ÈM [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚78373058651648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _step‚168 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1467176608256 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37075595 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1467264636928 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 % _timestamp‚1.7845026847911966e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5359370 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚81928747608064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37065437 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚15860346 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚76862907266048 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1467176608256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚15860344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14562746 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1462575235025 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚81928747608064 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚80461465931776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚80461482971136 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1510151385600 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21205093 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5359112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9203634 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199019008 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1462661383607 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 _runtime‚ 838.854492232 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37075595 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21215249 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OBDež±œdviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14563007 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚76862882362368 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21205093 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚80461465931776 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚78372859632640 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚2099 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚15860344 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37065437 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21215249 ODeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5480462 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1500436240384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37911020 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚83792854166528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1500524261888 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚80156420476928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16219075 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184608256 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16219073 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9413847 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9413844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21691947 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚83792854166528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37921164 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚81221171099056 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21702089 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1500436240384 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362ßõÑÄ'´ ¾O QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚80156210964992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1495814907459 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚81221188138416 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5480462 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1500436240384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37911020 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚83792854166528 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1500524261888 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚80156420476928 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16219075 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184608256 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16219073 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9413847 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9413844 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21691947 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚83792854166528 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37921164 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚81221171099056 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21702089 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1500436240384 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362«¦¡70µ + _wandbruntime‚857  _runtime‚857“ßï• ¶ ÃM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21701845 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5482928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37938397 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1544966214656 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 _runtime‚ 857.824403964 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16226396 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚83831003711488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚81258043166816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5482677 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚81258060206176 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16226394 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9418137 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚82329784293376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚82329784293376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚78647640224768 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80192606439424 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚78647665128448 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37928239 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16226396 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1501202378752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1501114350080 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9418134 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37938397 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚83831003711488 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21712001 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1545148718592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚172 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21701845 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚82754447877249 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1501202378752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14900811 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21712001 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1496404710433 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 % _timestamp‚1.7845027037610264e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1501114350080 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80192813847040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚83830898643456 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚82329801332736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚82754551065191 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 trainer/global_step‚2149 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416n" ;Î&· ÈM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 % _timestamp‚1.7845027037610264e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1501114350080 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚80192813847040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚83830898643456 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚82329801332736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚82754551065191 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 trainer/global_step‚2149 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21701845 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5482928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37938397 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503936 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1544966214656 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 _runtime‚ 857.824403964 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16226396 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚83831003711488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚81258043166816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5482677 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚81258060206176 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16226394 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9418137 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚82329784293376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚82329784293376 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚78647640224768 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚80192606439424 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚78647665128448 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚37928239 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16226396 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1501202378752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1501114350080 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9418134 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37938397 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚83831003711488 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚21712001 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1545148718592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚172 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21701845 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚82754447877249 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1501202378752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14900811 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21712001 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1496404710433 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0¬¼tgÝ0¸ + _wandbruntime‚857  _runtime‚857u𣹠ÿ /loss/FBt1/detection_loss‚11.735624362553809 /loss/FBt2/detection_loss‚1.5042588100859036 epoch‚2 % _timestamp‚1.7845027037627256e+09 _step‚173 0loss/FRt1/assignment_loss‚3.4238527306388407 2loss/SRqqt1/assignment_loss‚1.6499290618706832 2loss/SRqqt2/assignment_loss‚1.0879659156443084 /loss/FRt1/detection_loss‚0.6007551083089236 /loss/FRt2/detection_loss‚1.4200292629649496 /loss/FRt2/assignment_loss‚2.047289818230242 1loss/FBt1/assignment_loss‚0.08352841950698218 &loss/total_loss‚26.432636584343896 trainer/global_step‚2149 0loss/FBt2/assignment_loss‚0.2623435062869703 1loss/SRqqt1/detection_loss‚1.9232803300077952 1loss/SRqqt2/detection_loss‚0.6937792582444899 _runtime‚ 857.8256565271Ôy º „ 0loss/FBt2/assignment_loss‚0.2623435062869703 1loss/SRqqt1/detection_loss‚1.9232803300077952 1loss/SRqqt2/detection_loss‚0.6937792582444899 _runtime‚ 857.825656527 /loss/FBt1/detection_loss‚11.735624362553809 /loss/FBt2/detection_loss‚1.5042588100859036 epoch‚2 % _timestamp‚1.7845027037627256e+09 _step‚173 0loss/FRt1/assignment_loss‚3.4238527306388407 2loss/SRqqt1/assignment_loss‚1.6499290618706832 2loss/SRqqt2/assignment_loss‚1.0879659156443084 /loss/FRt1/detection_loss‚0.6007551083089236 /loss/FRt2/detection_loss‚1.4200292629649496 /loss/FRt2/assignment_loss‚2.047289818230242 1loss/FBt1/assignment_loss‚0.08352841950698218 &loss/total_loss‚26.432636584343896 trainer/global_step‚2149­#H½›» ‚(: »³õÒ“«À gpu.0.gpu‚65.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚113.785' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.92833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0܉è-i¼ ‚(:Þ »³õÒ╵À% disk./.usageGB‚1165.5418891906738! disk.nvme1c1n1.in‚ 10.48046875' disk.nvme1c1n1.out‚1077.97119140625& proc.memory.availableMB‚ 66.3203125 proc.cpu.threads‚17 disk.md0.out‚ 25.1328125 disk.dm-0.out‚7462.767578125 disk.rbd0.out‚0& disk.nvme0c0n1.out‚6713.3759765625 disk.nvme0n1.in‚ 48.3671875$ disk.nvme0n1.out‚6713.3759765625# proc.memory.rssMB‚ 2872.77734375) proc.memory.percent‚17.53404140472412 disk.md0.in‚0 disk.dm-0.in‚ 58.84765625! disk.dm-1.out‚274.99560546875 network.sent‚ 2.607063e+06 network.recv‚289828) disk./.usagePercent‚75.91870309009437 disk.nvme1n1.in‚ 10.48046875% disk.nvme1n1.out‚1077.97119140625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 48.3671875$ memory_percent‚99.59521293640137 cpu‚26.52420688705911Ìð–’0½ + _wandbruntime‚876  _runtime‚876óìÓȾ  % _timestamp‚1.7845027222606113e+09 _runtime‚ 876.323759849 _step‚174 ' lr-AdamW/pg1‚0.00011177638236723036 ' lr-AdamW/pg2‚0.00011177638236723036 trainer/global_step‚2199GB"ÈÍ¿ Ç _step‚174 ' lr-AdamW/pg1‚0.00011177638236723036 ' lr-AdamW/pg2‚0.00011177638236723036 trainer/global_step‚2199 % _timestamp‚1.7845027222606113e+09 _runtime‚ 876.323759849®½Ø¨Ÿ0À + _wandbruntime‚876  _runtime‚876È+áʾ'Á ¸O KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38783966 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9628344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84160753327104 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚85695110269952 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22198841 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚80396702929920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1529558234397 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22188699 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1579350897664 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85695110269952 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1529644382867 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16585125 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16585125 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184608256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚84594332721453 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚85695215330816 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16585123 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22198841 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚84160753327104 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38773822 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15231723 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9628347 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2199 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22188699 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚81976078731264 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚83064791526416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5603379 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚81975869219328 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1534462003712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16585123 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 _runtime‚ 876.32450942 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1534462003712% °õ­  ½O QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22188699 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1579350897664 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85695110269952 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1529644382867 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16585125 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16585125 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184608256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚84594332721453 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚85695215330816 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16585123 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22198841 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚84160753327104 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚38773822 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15231723 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9628347 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚2199 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22188699 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚81976078731264 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚83064791526416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5603379 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚81975869219328 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1534462003712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16585123 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 _runtime‚ 876.32450942 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1534462003712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚38783966 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9628344 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚84160753327104 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚85695110269952 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22198841 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚80396702929920 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1529558234397 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360¯Å#$0à + _wandbruntime‚876  _runtime‚876ŸlgÉ&Ä ÃM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15238535 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚80432398087168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84198102654976 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184601088 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84631983928599 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5606161 _step‚176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16592446 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚85733154746880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38791041 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚83101646554816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1580050028032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82012473018880 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 % _timestamp‚1.7845027225165327e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22198597 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9632634 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1535052091904 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38791041 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9632637 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16592444 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚84198119694336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1535140120576 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5605901 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚80432422990848 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84631880740657 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1535140120576 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15238798 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1530234185841 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38801199 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚83101663594176 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚85733259814912 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1530320334423 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712íe:Î&Å ÈM ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5606161 _step‚176 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16592446 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚85733154746880 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚38791041 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚83101646554816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1580050028032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82012473018880 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 % _timestamp‚1.7845027225165327e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22198597 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9632634 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1535052091904 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚38791041 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9632637 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16592444 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚84198119694336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1535140120576 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5605901 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚80432422990848 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84631880740657 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1535140120576 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15238798 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1530234185841 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚38801199 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚83101663594176 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚85733259814912 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1530320334423 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15238535 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚80432398087168 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84198102654976 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184601088 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84631983928599 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1563473858275 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16951173 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚82181460792320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 =6DeviceSD‚“-!tatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚86029071688704 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16951173 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22695593 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1568399745536 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9842844 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚86471868772691 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1568399745536 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22695593 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15569519 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 trainer/global_step‚2249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚86029071688704 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚83795787515392 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚82181485696000 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _runtime‚ 895.040400703 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16951175 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15569770 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚84908377875056 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845027409773207e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚86029054649344 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚87597366373376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1568311724032 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39646768 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9842847 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22685451 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16951175 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186705408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5726923 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚84908394914416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696"&DõÄ'Ð ¾O GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1568399745536 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚9842844 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚86471868772691 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1568399745536 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚22695593 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15569519 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 trainer/global_step‚2249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚86029071688704 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚83795787515392 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚82181485696000 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _runtime‚ 895.040400703 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚16951175 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15569770 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚84908377875056 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845027409773207e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚86029054649344 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚87597366373376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1568311724032 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚39646768 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚9842847 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22685451 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16951175 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186705408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5726923 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚84908394914416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1614301819392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1563473858275 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚16951173 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚82181460792320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚86029071688704 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16951173 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22695593 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330³7ËÐj0Ñ + _wandbruntime‚895  _runtime‚895}ˆÄ¹É&Ò ÃM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22705505 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚83832182523904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚86509416792007 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚16958496 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1568989833728 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39664001 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16958494 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚86066438055936 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1615001670656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚84945266982176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1568989833728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1564149809831 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚83831968824832 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5729334 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚86066421016576 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9847134 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚87635515918336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15576471 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1569077862400 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚87635515918336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82217180853248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39664001 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39653843 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 _step‚180 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86066438055936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22705505 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188795392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39653843 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15576226 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1564063661249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚86066421016576 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 % _timestamp‚1.7845027412293203e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚ 895.292593208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚2249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16958496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9847137 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚87635410850304 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22695349 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5729092 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22695349 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5729092 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚84945266982176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1568989833728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1564149809831 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚83831968824832 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 UIDeviceStatsMonitor.on_train_batch2pØÂÛ_end/inactive_split.small_pool.allocated‚5729334 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚86066421016576 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9847134 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚87635515918336 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15576471 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1569077862400 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚87635515918336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82217180853248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39664001 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚39653843 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 _step‚180 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86066438055936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068032 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22705505 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188795392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39653843 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15576226 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1564063661249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚86066421016576 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 % _timestamp‚1.7845027412293203e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚ 895.292593208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 trainer/global_step‚2249 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16958496 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9847137 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚87635410850304 ODeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚183 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚85615440141824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚88349301636099 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40499426 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚89499622476800 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚87897390050304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1597217185213 _runtime‚ 913.129597645 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚89499727537664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17317223 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚87897373010944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1597303333683 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15906946 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚87897373010944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1602249465856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5849602 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17317225 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1602337487360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15907211 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1602337487360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1602249465856 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚87897390050304 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚89499727537664 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚85615220144128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88349198448269 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚183 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚85615440141824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚88349301636099 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚40499426 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚89499622476800 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚87897390050304 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1597217185213 _runtime‚ 913.129597645 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚89499727537664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17317223 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚87897373010944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1597303333683 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15906946 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚87897373010944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1602249465856 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5849602 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17317225 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1602337487360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15907211 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1602337487360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1602249465856 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚87897390050304 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚89499727537664 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚85615220144128 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚88349198448269 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 I6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚87934739378176 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5851947 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309632 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1602927575552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88386746467473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5852196 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1649703969280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40516645 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_bçÀRGåatch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚87934739378176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1597893136657 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40526803 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15913833 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1597979285239 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚86788870370176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚86788853330816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚89537772021760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7845027593172388e+09 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚88386849655415 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23202257 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40516645 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚89537666953728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚84001938715648 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17324544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15913581 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚87934756417536 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1603015604224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1603015604224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10061634 _runtime‚ 913.380581453DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚87934739378176 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1597893136657 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚40526803 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15913833 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1597979285239 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚86788870370176 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚86788853330816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚89537772021760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7845027593172388e+09 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚88386849655415 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚23202257 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚40516645 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚89537666953728 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚84001938715648 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17324544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15913581 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚87934756417536 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1603015604224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1603015604224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10061634 _runtime‚ 913.380581453 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10061637 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚87934739378176 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5851947 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309632 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1602927575552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88386746467473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5852196 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1649703969280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚40516645 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 E0ì + _wandbruntime‚931  _runtime‚931ÕzònÁ'í »O IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚5973848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1684161953280 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚88595601690416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1636275229184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚2349 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16245692 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17683273 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23689097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85751001420800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚89765691372544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚89765708411904 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5974105 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87434953862144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85750976517120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1683977345024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10271847 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184608256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚90226734499507 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 % _timestamp‚1.7845027774786189e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23678955 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1636187207680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚90226631311677 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚91401878580224 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚89765708411904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚87435163374080 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚91401878580224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚91401983641088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10271844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1631132809091 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1631046660621 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚89765691372544 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1636187207680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1636275229184 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41372372 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17683273 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23689097 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23689097 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚85751001420800 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚89765691372544 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚89765708411904 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105060864 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5974105 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚87434953862144 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚85750976517120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1683977345024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10271847 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184608256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚90226734499507 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 % _timestamp‚1.7845027774786189e+09 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23678955 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1636187207680 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚90226631311677 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚91401878580224 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚89765708411904 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚87435163374080 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚91401878580224 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚91401983641088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10271844 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1631132809091 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1631046660621 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10142 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚89765691372544 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1636187207680 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88021504 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10144 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1636275229184 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚41372372 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17683273 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23689097 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313565696»[2…0ï + _wandbruntime‚931  _runtime‚931AËdïË&ð ÅM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚89803057739776 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚87471347763712 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1631808760647 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23699009 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41379447 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚88632456718816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚91439923057152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5976513 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85786696578048 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚89803057739776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10156 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12373 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16252650 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚87471559365632 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚89803074779136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10276134 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚88632473758176 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17690594 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91440028125184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41379447 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1631722612065 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1636865317376 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186698240 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚90264179330881 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17690596 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41389605 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1684676089344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16252397 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1684862787584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23688853 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚89803074779136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1636865317376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17690596 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 _step‚188 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚89803074779136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10276134 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚88632473758176 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12373 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17690594 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601920 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91440028125184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41379447 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1631722612065 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1636865317376 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186698240 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚90264179330881 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17690596 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41389605 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1684676089344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16252397 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1684862787584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23688853 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚89803074779136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1636865317376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17690596 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 _step‚188 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚91439923057152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10158 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5976513 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚85786696578048¼Tì÷N0ò + _wandbruntime‚931  _runtime‚9319Ò½ó ú 0loss/FBt1/assignment_loss‚0.3279383200410156 .loss/FBt1/detection_loss‚1.876085055614545 1loss/SRqqt2/assignment_loss‚1.060207905312755 /loss/FBt2/detection_loss‚1.7425159031665274 %loss/total_loss‚16.06716735060366 epoch‚2 trainer/global_step‚2349 _runtime‚ 931.799049674 _step‚189 /loss/FRt1/assignment_loss‚3.345847804523265 2loss/SRqqt1/assignment_loss‚1.3415819972249694 /loss/FRt1/detection_loss‚0.5838423799630367 /loss/FRt2/detection_loss‚1.3360447655004601 1loss/SRqqt2/detection_loss‚0.7861400673086989 0loss/FRt2/assignment_loss‚1.7940341983316888 0loss/FBt2/assignment_loss‚0.1805850339883427 0loss/SRqqt1/detection_loss‚1.692343919628356 % _timestamp‚1.7845027777362082e+09“‘lžô ÿ %loss/total_loss‚16.06716735060366 epoch‚2 trainer/global_step‚2349 _runtime‚ 931.799049674 _step‚189 /loss/FRt1/assignment_loss‚3.345847804523265 2loss/SRqqt1/assignment_loss‚1.3415819972249694 /loss/FRt1/detection_loss‚0.5838423799630367 /loss/FRt2/detection_loss‚1.3360447655004601 1loss/SRqqt2/detection_loss‚0.7861400673086989 0loss/FRt2/assignment_loss‚1.7940341983316888 0loss/FBt2/assignment_loss‚0.1805850339883427 0loss/SRqqt1/detection_loss‚1.692343919628356 % _timestamp‚1.7845027777362082e+09 0loss/FBt1/assignment_loss‚0.3279383200410156 .loss/FBt1/detection_loss‚1.876085055614545 1loss/SRqqt2/assignment_loss‚1.060207905312755 /loss/FBt2/detection_loss‚1.7425159031665274½3>$nõ ‚(:ã †´õÒ„Çç¿ disk.md0.in‚0$ memory_percent‚99.59495067596436) disk./.usagePercent‚76.09855727640043 disk.dm-0.in‚ 58.84765625 disk.dm-1.in‚0! disk.dm-1.out‚275.71923828125) proc.memory.percent‚17.53404140472412 proc.cpu.threads‚17 network.recv‚310405 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 48.3671875 disk.nvme0n1.in‚ 48.3671875! disk.nvme1c1n1.in‚ 10.48046875 disk.md0.out‚ 26.6640625' proc.memory.availableMB‚ 66.36328125# proc.memory.rssMB‚ 2872.77734375 network.sent‚ 2.848609e+06% disk./.usageGB‚1168.3031005859375" disk.dm-0.out‚9424.85302734375 disk.rbd0.out‚0' disk.nvme1c1n1.out‚1081.69775390625 disk.nvme1n1.in‚ 10.48046875 cpu‚26.48455163458593' disk.nvme0c0n1.out‚8675.82373046875% disk.nvme0n1.out‚8675.82373046875% disk.nvme1n1.out‚1081.69775390625.žY¦›ö ‚(: †´õÒ­¡šÀ gpu.0.gpu‚31.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.767' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.25566666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yF‡+0÷ + _wandbruntime‚950  _runtime‚950ùŒ3Èø  ' lr-AdamW/pg1‚0.00011166902821201419 ' lr-AdamW/pg2‚0.00011166902821201419 trainer/global_step‚2399 % _timestamp‚1.7845027961993291e+09 _runtime‚ 950.262464282 _step‚190öîJÍù Ç ' lr-AdamW/pg1‚0.00011166902821201419 ' lr-AdamW/pg2‚0.00011166902821201419 trainer/global_step‚2399 % _timestamp‚1.7845027961993291e+09 _runtime‚ 950.262464282 _step‚190¾c1AÁ0ú + _wandbruntime‚950  _runtime‚950ÍOÇÄû »O JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16583653 =6DeviceStatsMonitor.on_train_batãdäœý&ch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24175707 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚87535759283200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1670212971008 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6097309 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1719093542400 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89254852825600 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18049323 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1670212971008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87535734379520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚93304134683648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90439188039056 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚91634009734144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93304134683648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18049323 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚89254639119360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10486344 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _step‚191 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18049325 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1670124949504 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16583911 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213706240 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1664876136029 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚91634009734144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1670124949504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 trainer/global_step‚2399 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 _runtime‚ 950.263248313 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24185849 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚91634026773504 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚92104064175085 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24175707 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90439205078416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42235174 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚92104167362915ôÖÃÆÆ'ü ÀO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚16583653 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24175707 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12302 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚87535759283200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1670212971008 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6097309 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1719093542400 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105060864 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚89254852825600 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18049323 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1670212971008 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12373 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12373 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87535734379520 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚93304134683648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181451 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90439188039056 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚91634009734144 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚93304134683648 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18049323 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚89254639119360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682747 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10486344 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _step‚191 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313565696 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18049325 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1670124949504 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚16583911 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10142 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213706240 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88021504 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1664876136029 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚91634009734144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1670124949504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 trainer/global_step‚2399 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12302 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187830 _runtime‚ 950.263248313 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24185849 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148470 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚91634026773504 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚92104064175085 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚24175707 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90439205078416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚42235174 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚92104167362915¿_0ý + _wandbruntime‚950  _runtime‚950ËÁôÊ&þ ÄM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1670803059200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 % _timestamp‚1.7845027964519491e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16590660 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24195761 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚93342179160576 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42242249 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6100023 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚89291035268608 _step‚192 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚93342284228608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1665638236055 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1665552087473 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10490637 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1719788235776 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚90476060106816 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16590419 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1670891087872 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚93342284228608 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24185605 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1670803059200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1670891087872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 trainer/global_step‚2399 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24195761 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18056646 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10490634 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18056646 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18056644 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42252407 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚90476077146176;˜²ì ÿ ÉM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚90476077146176 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1670803059200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 % _timestamp‚1.7845027964519491e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16590660 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682747 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚24195761 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚93342179160576 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚42242249 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6100023 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚89291035268608 _step‚192 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚93342284228608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88028672 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1665638236055 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1665552087473 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10490637 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187942 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148582 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407616 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1719788235776 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚90476060106816 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16590419 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068032 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1670891087872 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚93342284228608 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚24185605 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88028672 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10156 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313565696 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1670803059200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1670891087872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313565696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 trainer/global_step‚2399 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚24195761 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12302 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18056646 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10490634 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18056646 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18056644 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚42252407 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181451 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330À݃;‰0€ + _wandbruntime‚950  _runtime‚950 E)œ ú 0loss/SRqqt2/assignment_loss‚1.03955670789314 _step‚193 /loss/FRt1/assignment_loss‚3.291150990435831 0loss/FRt2/assignment_loss‚1.8202702151891779 1loss/FBt2/assignment_loss‚0.05632241528772397 .loss/FBt2/detection_loss‚1.168261513623071 &loss/total_loss‚15.079964688271463 trainer/global_step‚2399 % _timestamp‚1.7845027964528375e+09 2loss/SRqqt1/assignment_loss‚1.2211177181452513 /loss/FRt1/detection_loss‚0.5667202776081288 /loss/FBt1/detection_loss‚1.3942847647594638 0loss/FBt1/assignment_loss‚0.2768670388212832 /loss/FRt2/detection_loss‚1.4435086203326337 0loss/SRqqt1/detection_loss‚2.045888467878103 0loss/SRqqt2/detection_loss‚0.756015958297655 epoch‚2 _runtime‚ 950.515740101;0¤‚ ÿ /loss/FRt1/assignment_loss‚3.291150990435831 0loss/FRt2/assignment_loss‚1.8202702151891779 1loss/FBt2/assignment_loss‚0.05632241528772397 .loss/FBt2/detection_loss‚1.168261513623071 &loss/total_loss‚15.079964688271463 trainer/global_step‚2399 % _timestamp‚1.7845027964528375e+09 2loss/SRqqt1/assignment_loss‚1.2211177181452513 /loss/FRt1/detection_loss‚0.5667202776081288 /loss/FBt1/detection_loss‚1.3942847647594638 0loss/FBt1/assignment_loss‚0.2768670388212832 /loss/FRt2/detection_loss‚1.4435086203326337 0loss/SRqqt1/detection_loss‚2.045888467878103 0loss/SRqqt2/detection_loss‚0.756015958297655 epoch‚2 _runtime‚ 950.515740101 0loss/SRqqt2/assignment_loss‚1.03955670789314 _step‚193ÁïÈðokƒ ‚(:à •´õÒ—Þö¿' disk.nvme0c0n1.out‚8701.45654296875% disk.nvme0n1.out‚8701.45654296875 disk.nvme1n1.in‚ 10.48046875 network.recv‚313647% disk.nvme1n1.out‚1082.70947265625 disk.md0.in‚0 disk.dm-0.in‚ 58.84765625# memory_percent‚99.5978593826294 proc.memory.rssMB‚ 2873.4375* proc.memory.percent‚17.538070678710938 network.sent‚ 2.92351e+06 disk.nvme0n1.in‚ 48.3671875" disk.dm-0.out‚9450.13037109375' proc.memory.availableMB‚ 65.88671875 proc.cpu.threads‚17! disk.dm-1.out‚275.85205078125 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme1c1n1.in‚ 10.48046875 disk.md0.out‚ 27.24609375 cpu‚26.535087637626788' disk.nvme1c1n1.out‚1082.70947265625) disk./.usagePercent‚76.09855727640043% disk./.usageGB‚1168.3031005859375 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 48.3671875è\!›„ ‚(: •´õÒà™£À gpu.0.gpu‚43.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚152.519' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚50.83966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó()0… + _wandbruntime‚968  _runtime‚968ýX2d>† Œ| ("Purity/0FRt1SRqqt2FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_3‚0.7857940196990967 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7802366614341736 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.23389387130737305 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 +jet/accuracy_3_of_4‚0.02901635505259037 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.11156313866376877 6validation_average_jet_accuracy‚0.2833218276500702 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.30000001192092896 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3571428656578064 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.1875 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5409555435180664 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9963423609733582 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.30422529578208923 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004007973708212376 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.4627193212509155 ) Purity/0FRt1SRqqt2FBt/FBt_purity‚0.75 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9915291666984558 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16336236894130707 'particle/f_score‚0.5785264372825623 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.42222222685813904 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.29098188877105713 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +particle/sensitivity‚0.5568750500679016 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 jet/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4890635311603546 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9753512144088745 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9326364398002625 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17747478187084198 particle/accuracy_2_of_4‚1 particle/accuracy_3_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5107592344284058 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.940365731716156 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5187895894050598 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15366607904434204 *loss/val_total_loss‚16.342646780740257 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2193065881729126 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.22727273404598236 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5447795987129211 *jet/accuracy_2_of_3‚0.3293350338935852 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1832631379365921 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.061933428049087524 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.3294526934623718 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7046993374824524 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.14827114343643188 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.06310183554887772 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4677881598472595 +jet/accuracy_4_of_4‚0.00833333283662796 epoch‚2 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23986166715621948 *jet/accuracy_1_of_1‚0.2443622201681137 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.49795418977737427 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13537415862083435 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_5_of_5‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9758173823356628 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.49273785948753357 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.22727273404598236 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16336236894130707 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9611111283302307 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11157797276973724 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9884267449378967 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1612037867307663 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 4 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1881113499403 (particle/accuracy‚0.8257843255996704 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4678456783294678 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5104639530181885 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_3‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.14003539085388184 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.0993054062128067 +particle/specificity‚0.8993210792541504 % _timestamp‚1.7845028139412544e+09 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9832484126091003 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8416918516159058 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.38849589228630066 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.47777244448661804 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 _step‚194 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6411803364753723 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4664055407047272 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4565395712852478 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01358194462954998 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2142857164144516 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3727591037750244 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9707067608833313 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.13624699413776398 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2954545319080353 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.04270833730697632 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.1458333283662796 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9982993602752686 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.11627346277236938 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9807692170143127 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4953186511993408 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5574040412902832 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.39998722076416016 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.007262995466589928 particle/accuracy_1_of_4‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9660088419914246 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9812713861465454 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.18470589816570282 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 /particle/accuracy_4_of_4‚0.9124244451522827 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2993827164173126 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9772435426712036 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09775964170694351 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2362208068370819 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.762499988079071 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.28746020793914795 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5304297208786011 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9799532294273376 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9898989200592041 particle/accuracy_2_of_3‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4375 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18605226278305054 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.39600855112075806 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9882937073707581 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.358759343624115 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.399000346660614 particle/accuracy_1_of_1‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3576388657093048 particle/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.11627346277236938 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6932539939880371 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.12668223679065704 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 +jet/accuracy_3_of_3‚0.04070870950818062 trainer/global_step‚2402 _runtime‚ 968.004841717 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.48064354062080383 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2116188257932663 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0335417166352272 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.49273785948753357 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5652391910552979 jet/accuracy_4_of_5‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9951894879341125 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3577577769756317 4 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9903564453125 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.08930432796478271 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7887948155403137 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3030302822589874 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9943211674690247 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.23558568954467773 jet/accuracy_2_of_5‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9923917055130005 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.8896373510360718 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5850358009338379 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 /particle/accuracy_3_of_3‚0.9985008239746094 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5407511591911316 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.2750000059604645 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.30000001192092896 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3571428656578064 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.7091541290283203 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.44371578097343445 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.441032737493515 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9528927803039551 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9841521978378296 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ( Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.28918734192848206 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.2854166626930237 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.11477934569120407 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4082646071910858 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5309647917747498 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.007851792499423027 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5304526686668396 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2117270976305008 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19568589329719543 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9951894879341125 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.49239230155944824 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12446204572916031 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5837056636810303 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9952380657196045 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4295273423194885 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7072649598121643 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7266755104064941 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8350493311882019 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09068574011325836 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1281571388244629 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 *jet/accuracy_1_of_2‚0.6779966354370117 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.16664087772369385 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.3516865074634552 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13738210499286652 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.37173333764076233 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9448186159133911 .particle/accuracy_3_of_4‚0.986640214920044 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7092105746269226 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1466488391160965 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3294526934623718 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.49557921290397644 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23751822113990784 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.800000011920929 jet/accuracy_3_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.027855584397912025 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5277692079544067 particle/accuracy_1_of_2‚1XrïF•‡ ‘| ( Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.28918734192848206 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.2854166626930237 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.11477934569120407 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4082646071910858 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5309647917747498 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.007851792499423027 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5304526686668396 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.2117270976305008 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 particle/accuracy_1_of_5‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19568589329719543 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9951894879341125 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.49239230155944824 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12446204572916031 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5837056636810303 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9952380657196045 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4295273423194885 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7072649598121643 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7266755104064941 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8350493311882019 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09068574011325836 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1281571388244629 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 *jet/accuracy_1_of_2‚0.6779966354370117 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.16664087772369385 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 *jet/accuracy_2_of_4‚0.3516865074634552 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8333333134651184 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13738210499286652 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.37173333764076233 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9448186159133911 .particle/accuracy_3_of_4‚0.986640214920044 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7092105746269226 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1466488391160965 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.3294526934623718 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.49557921290397644 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23751822113990784 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.800000011920929 jet/accuracy_3_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.027855584397912025 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5277692079544067 particle/accuracy_1_of_2‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_3‚0.7857940196990967 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7802366614341736 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.23389387130737305 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 +jet/accuracy_3_of_4‚0.02901635505259037 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.11156313866376877 6validation_average_jet_accuracy‚0.2833218276500702 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.30000001192092896 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3571428656578064 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.1875 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.5409555435180664 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9963423609733582 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.30422529578208923 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004007973708212376 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.4627193212509155 ) Purity/0FRt1SRqqt2FBt/FBt_purity‚0.75 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9915291666984558 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16336236894130707 'particle/f_score‚0.5785264372825623 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.42222222685813904 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.29098188877105713 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +particle/sensitivity‚0.5568750500679016 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 jet/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4890635311603546 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9753512144088745 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9326364398002625 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.17747478187084198 particle/accuracy_2_of_4‚1 particle/accuracy_3_of_5‚1 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5107592344284058 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.940365731716156 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.5187895894050598 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15366607904434204 *loss/val_total_loss‚16.342646780740257 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2193065881729126 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.22727273404598236 @&Purity/2FRt*SRqqt1FBt/event_proportioÐ7`‚$n‚0.0016043527284637094 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5447795987129211 *jet/accuracy_2_of_3‚0.3293350338935852 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1832631379365921 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.061933428049087524 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.3294526934623718 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7046993374824524 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.14827114343643188 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.06310183554887772 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4677881598472595 +jet/accuracy_4_of_4‚0.00833333283662796 epoch‚2 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.23986166715621948 *jet/accuracy_1_of_1‚0.2443622201681137 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.49795418977737427 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13537415862083435 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_5_of_5‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9758173823356628 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.49273785948753357 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.22727273404598236 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.16336236894130707 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9611111283302307 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11157797276973724 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9884267449378967 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1612037867307663 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 4 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1881113499403 (particle/accuracy‚0.8257843255996704 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4678456783294678 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5104639530181885 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_1_of_3‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.14003539085388184 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.0993054062128067 +particle/specificity‚0.8993210792541504 % _timestamp‚1.7845028139412544e+09 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9832484126091003 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8416918516159058 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.38849589228630066 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.47777244448661804 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 _step‚194 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6411803364753723 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4664055407047272 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4565395712852478 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.01358194462954998 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2142857164144516 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.3727591037750244 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9707067608833313 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.13624699413776398 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.2954545319080353 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.04270833730697632 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.1458333283662796 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9982993602752686 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.11627346277236938 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9807692170143127 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 jet/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4953186511993408 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5574040412902832 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.39998722076416016 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.007262995466589928 particle/accuracy_1_of_4‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9660088419914246 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9812713861465454 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.18470589816570282 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 /particle/accuracy_4_of_4‚0.9124244451522827 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.2993827164173126 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9772435426712036 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09775964170694351 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.2362208068370819 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.762499988079071 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.28746020793914795 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5304297208786011 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9799532294273376 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9898989200592041 particle/accuracy_2_of_3‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.4375 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.18605226278305054 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.39600855112075806 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9882937073707581 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.358759343624115 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.399000346660614 particle/accuracy_1_of_1‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3576388657093048 particle/accuracy_4_of_5‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.11627346277236938 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.6932539939880371 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.12668223679065704 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 +jet/accuracy_3_of_3‚0.04070870950818062 trainer/global_step‚2402 _runtime‚ 968.004841717 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.48064354062080383 7 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.2116188257932663 9"Purity/1FRt1SRqqt*FBt/event_purity‚0.0335417166352272 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.49273785948753357 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5652391910552979 jet/accuracy_4_of_5‚0 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9951894879341125 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.3577577769756317 4 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9903564453125 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.08930432796478271 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7887948155403137 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3030302822589874 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9943211674690247 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.23558568954467773 jet/accuracy_2_of_5‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9923917055130005 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.8896373510360718 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5850358009338379 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 /particle/accuracy_3_of_3‚0.9985008239746094 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5407511591911316 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.2750000059604645 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.30000001192092896 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3571428656578064 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.7091541290283203 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.44371578097343445 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.441032737493515 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9528927803039551 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9841521978378296 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505ÂÁ›bœjˆ ‚(:ß ¤´õÒÁÁ¨À! disk.nvme1c1n1.in‚ 10.48046875 network.recv‚318101$ disk./.usageGB‚1168.256763458252 disk.md0.in‚0 disk.dm-0.in‚ 58.84765625 disk.nvme0c0n1.in‚ 48.3671875# disk.nvme0n1.out‚9449.388671875% disk.nvme1n1.out‚1271.08544921875$ memory_percent‚99.20732975006104 disk.dm-1.in‚0 disk.rbd0.in‚0% disk.nvme0c0n1.out‚9449.388671875 disk.nvme1n1.in‚ 10.48046875( proc.memory.availableMB‚ 129.87109375* proc.memory.percent‚17.537713050842285 cpu‚26.581800821719874 proc.cpu.threads‚17# disk.dm-0.out‚10321.26708984375 disk.dm-1.out‚339.8076171875 disk.rbd0.out‚0' disk.nvme1c1n1.out‚1271.08544921875# proc.memory.rssMB‚ 2873.37890625 network.sent‚ 2.985055e+06( disk./.usagePercent‚76.0955390625795 disk.md0.out‚ 27.828125 disk.nvme0n1.in‚ 48.3671875Öë p›‰ ‚(: ¤´õÒÀöÒÀ gpu.0.gpu‚62.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚200.288' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.76266666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0{£Ñ0Š + _wandbruntime‚986  _runtime‚986ü›µÆ‹ À % _timestamp‚1.7845028322274714e+09 _runtime‚ 986.290576231 _step‚195 & lr-AdamW/pg1‚0.0001116399619746568 & lr-AdamW/pg2‚0.0001116399619746568 trainer/global_step‚24499æ‚ËŒ Å & lr-AdamW/pg1‚0.0001116399619746568 & lr-AdamW/pg2‚0.0001116399619746568 trainer/global_step‚2449 % _timestamp‚1.7845028322274714e+09 _runtime‚ 986.290576231 _step‚195Ã"½0 + _wandbruntime‚986  _runtime‚9867ÏLÁ'Ž »O @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 trainer/global_step‚2449 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚94475008730112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚94474991690752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚96199474698240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201122816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _step‚196 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24849866 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18652843 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24849866 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96199474698240 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚96199579759616 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24839723 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 trainer/global_step‚2449 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚94475008730112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚94474991690752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚96199474698240 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201122816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _step‚196 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚24849866 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18652843 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚24849866 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚96199474698240 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚96199579759616 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚24839723 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2449 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43519942 ODeviceStatsMonitor.on_t´ó(èF"rain_batch_end/active_bytes.all.current‚ 105068544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _runtime‚ 986.542550329 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚93288170002528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚92056660426752 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90281635566592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17101579 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1775228072960 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1725161117184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18660162 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚93288152963168 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18660164 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90281610662912 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 $ _timestamp‚1.784502832479305e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24849621 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1775049763840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚94512375097344 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚96237519175168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 _step‚197 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43509783 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10799214 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6302365 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43519942 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18660164 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚92056863639552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0ìõyoÏ&’ ÉM E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17101579 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1775228072960 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1725161117184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18660162 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚93288152963168 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18660164 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90281610662912 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 $ _timestamp‚1.784502832479305e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚24849621 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1775049763840 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚94512375097344 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚96237519175168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 _step‚197 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43509783 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10799214 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6302365 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43519942 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18660164 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚92056863639552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚2449 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43519942 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _runtime‚ 986.542550329 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚93288170002528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚92056660426752 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚90281635566592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0Å¡áÕd0“ + _wandbruntime‚986  _runtime‚986Œ\èÎý” ÷ /loss/FRt2/detection_loss‚1.3011464872716327 1loss/SRqqt1/detection_loss‚1.8820013762183057 _step‚198 .loss/FBt1/detection_loss‚2.261939614999574 %loss/total_loss‚16.14989274491861 trainer/global_step‚2449 % _timestamp‚1.7845028324802153e+09 0loss/FRt1/assignment_loss‚3.2795696698724504 1loss/SRqqt2/assignment_loss‚1.067623122482972 /loss/FRt1/detection_loss‚0.6488049154934591 .loss/FBt2/detection_loss‚1.390136777987209 epoch‚3 _runtime‚ 986.5430171 0loss/SRqqt1/assignment_loss‚1.47124463158588 0loss/FBt2/assignment_loss‚0.0722812843204963 1loss/SRqqt2/detection_loss‚0.7837648790006413 /loss/FRt2/assignment_loss‚1.796529775781996 1loss/FBt1/assignment_loss‚0.19485020990399562§ • ü /loss/FRt2/assignment_loss‚1.796529775781996 1loss/FBt1/assignment_loss‚0.19485020990399562 /loss/FRt2/detection_loss‚1.3011464872716327 1loss/SRqqt1/detection_loss‚1.8820013762183057 _step‚198 .loss/FBt1/detection_loss‚2.261939614999574 %loss/total_loss‚16.14989274491861 trainer/global_step‚2449 % _timestamp‚1.7845028324802153e+09 0loss/FRt1/assignment_loss‚3.2795696698724504 1loss/SRqqt2/assignment_loss‚1.067623122482972 /loss/FRt1/detection_loss‚0.6488049154934591 .loss/FBt2/detection_loss‚1.390136777987209 epoch‚3 _runtime‚ 986.5430171 0loss/SRqqt1/assignment_loss‚1.47124463158588 0loss/FBt2/assignment_loss‚0.0722812843204963 1loss/SRqqt2/detection_loss‚0.7837648790006413Æ-QÝd– ‚(:Ù ³´õÒ‚ñ¿% disk./.usageGB‚1168.6652183532715 disk.md0.out‚ 28.1953125 disk.dm-0.in‚ 58.84765625' disk.nvme0c0n1.out‚9462.47998046875" disk.nvme1c1n1.out‚ 1335.703125# proc.memory.rssMB‚ 2873.37890625* proc.memory.percent‚17.537713050842285 proc.cpu.threads‚17) disk./.usagePercent‚76.12214416891526 disk.nvme1n1.in‚ 10.48046875 disk.md0.in‚0 disk.nvme0c0n1.in‚ 48.3671875 disk.nvme0n1.in‚ 48.3671875& proc.memory.availableMB‚ 65.1796875 disk.nvme1n1.out‚ 1335.703125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚9462.47998046875! disk.nvme1c1n1.in‚ 10.48046875 cpu‚26.59180740824809 network.sent‚ 3.029227e+06 network.recv‚321880# disk.dm-0.out‚10334.28759765625 disk.dm-1.out‚403.7060546875$ memory_percent‚99.60217475891113Cü#G— ‚(:… ³´õÒ„åÆÀ gpu.0.gpu‚56.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚113.628' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.876 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A7̺h˜ ‚(:Ý ´õÒ´Áà¿% disk.nvme0n1.out‚9466.37060546875' proc.memory.availableMB‚ 65.17578125 network.recv‚326232) disk./.usagePercent‚76.12214565975933$ disk./.usageGB‚1168.665241241455 disk.nvme1n1.in‚ 10.48046875 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 48.3671875 disk.nvme0n1.in‚ 48.3671875! disk.nvme1c1n1.in‚ 10.48046875 network.sent‚ 3.093372e+06& disk.nvme1c1n1.out‚1336.0888671875* proc.memory.percent‚17.537713050842285 proc.cpu.threads‚17 disk.md0.out‚28.375 disk.dm-0.in‚ 58.84765625 disk.rbd0.out‚0$ memory_percent‚99.60219860076904# proc.memory.rssMB‚ 2873.37890625 cpu‚26.603465230829496$ disk.nvme1n1.out‚1336.0888671875 disk.md0.in‚0# disk.dm-0.out‚10338.02587890625 disk.dm-1.out‚ 403.837890625 disk.rbd0.in‚0' disk.nvme0c0n1.out‚9466.37060546875ˆ=•Æ™ ‚(:’ ´õÒ•ÜŒÀ gpu.0.gpu‚91.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚118.797' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.599000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨šÊ‹2š - _wandbruntime‚1005  _runtime‚1005uZwDÉ› à ' lr-AdamW/pg1‚0.00011161000556758534 ' lr-AdamW/pg2‚0.00011161000556758534 trainer/global_step‚2499 % _timestamp‚1.7845028512152245e+09 _runtime‚1005.278362577 _step‚199‚­i•Îœ È ' lr-AdamW/pg1‚0.00011161000556758534 ' lr-AdamW/pg2‚0.00011161000556758534 trainer/global_step‚2499 % _timestamp‚1.7845028512152245e+09 _runtime‚1005.278362577 _step‚199ÇS[ ‘2 - _wandbruntime‚1005  _runtime‚1005©!‹_f$ž »O VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1758508771328 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚96343327091712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25336475 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1752929094406 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚98101835863040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚96343310052352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17432200 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93839986396672 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11009427 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚98101730801664 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44355366 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19018891 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19018891 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚92030673368064 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19018893 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11009424 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6422776 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1809493442048 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1758420749312 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95094884283408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25336475 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚96343327091712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44365511 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25346618 _step‚200 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚96343310052352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 $ _timestamp‚1.784502851216047e+09 _runtime‚1005.279130808 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1758420749312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_Xû}[batch_start/active.large_pool.allocated‚19018893 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17432434 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚92030698271744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚98101835863040 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98101730801664 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96847813377814 trainer/global_step‚2499 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1809313028608 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717ï*3@Æ'Ÿ ÀO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25336475 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚96343327091712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚44365511 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25346618 _step‚200 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚96343310052352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚234 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 $ _timestamp‚1.784502851216047e+09 _runtime‚1005.279130808 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1758420749312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19018893 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17432434 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚92030698271744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚98101835863040 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98101730801664 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96847813377814 trainer/global_step‚2499 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1809313028608 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1758508771328 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚96343327091712 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25336475 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1752929094406 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚98101835863040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚96343310052352 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17432200 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93839986396672 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11009427 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚98101730801664 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚44355366 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19018891 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19018891 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚92030673368064 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19018893 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11009424 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6422776 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1809493442048 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1758420749312 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95094884283408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104ÈÍ]2  - _wandbruntime‚1005  _runtime‚1005ÈHË&¡ ÅM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25346373 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚201 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1753605045850 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _runtime‚ 1005.53250815 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19026214 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1759098859008 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17439205 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17438947 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92066393428992 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44382744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98139880347136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚96380676419584 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1759186888192 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98139880347136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6425488 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6425233 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1753691194436 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96885361397018 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93876589284352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1810195855360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 trainer/global_step‚2499a›+,Ð&¢ ÊM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚95131756351168 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1759098859008 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚96380676419584 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17438947 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚92066393428992 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚44382744 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98139880347136 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚96380676419584 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1759186888192 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98139880347136 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6425488 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6425233 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1753691194436 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚96885361397018 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93876589284352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1810195855360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 trainer/global_step‚2499 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25346373 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚201 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1753605045850 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _runtime‚ 1005.53250815 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19026214 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1759098859008 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17439205 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98211645453312 trainer/global_step‚2549 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚100004091966464 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚98211645453312 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95659888894464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚98725246241222 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 _step‚204 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚98211628413952 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100003986905088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11223927 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17770118 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1786844718288 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25843370 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6546194 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚100004091966464 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1792358491136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚98211628413952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45228313 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25833227 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19384941 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 _runtime‚1024.145542989 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19384943 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93815431230464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚96938504710768 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1786758569814 % _timestamp‚1.7845028700812852e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248EÇ++Ë'­ ÅO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 _runtime‚1024.145542989 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19384943 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93815431230464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚96938504710768 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1786758569814 % _timestamp‚1.7845028700812852e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25843370 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚98725349429056 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1792358491136 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45218168 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19384943 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚98211645453312 trainer/global_step‚2549 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚100004091966464 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚98211645453312 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚95659888894464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚98725246241222 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 _step‚204 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚98211628413952 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚100003986905088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11223927 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17770118 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1786844718288 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25843370 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6546194 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚100004091966464 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1792358491136 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚98211628413952 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45228313 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚25833227 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19384941 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016ÌÎ&m2® - _wandbruntime‚1024  _runtime‚1024{8ágÐ&¯ ÊM BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1787434521258 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19392262 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1793124630016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19392262 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚98248994781184 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚98249011820544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45235387 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚100042136450560 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚96975376778528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25853282 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25843125 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1844949174272 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25843125 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19392264 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚98248994781184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1793124630016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11228217 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6548605 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93851126387712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45245546 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6548862 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93851151291392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 trainer/global_step‚2549 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1793036600832 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚95696282969088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 _step‚205 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17776819 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25853282 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚98762897448372Tå ¶° ÏM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚45235387 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚100042136450560 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E?DeviceStatsMonitor.on_train_batch_end/oversize_segp,K$ments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚96975376778528 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25853282 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25843125 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1844949174272 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25843125 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19392264 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚98248994781184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1793124630016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11228217 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6548605 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93851126387712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45245546 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6548862 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚93851151291392 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 trainer/global_step‚2549 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1793036600832 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚95696282969088 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 _step‚205 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17776819 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25853282 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚98762897448372 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1787434521258 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19392262 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1793124630016 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19392262 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚98248994781184 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚98249011820544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184ÍVV(N2± - _wandbruntime‚1024  _runtime‚1024À’œ² ü 2loss/SRqqt1/assignment_loss‚1.1986923530628835 1loss/SRqqt1/detection_loss‚2.0010126044055343 /loss/FRt1/detection_loss‚0.5249569383377258 /loss/FRt2/detection_loss‚1.4722987138094383 0loss/SRqqt2/detection_loss‚0.733762390693189 .loss/FBt1/detection_loss‚2.496926337826057 %loss/total_loss‚16.16530656934943 0loss/FRt2/assignment_loss‚1.7875649001396878 1loss/FBt1/assignment_loss‚0.11135419894313707 $ _timestamp‚1.784502870334971e+09 _runtime‚1024.397926292 epoch‚3 2loss/SRqqt2/assignment_loss‚0.9924125296095795 0loss/FBt2/assignment_loss‚0.1914710106329081 .loss/FBt2/detection_loss‚1.541821470213698 trainer/global_step‚2549 _step‚206 0loss/FRt1/assignment_loss‚3.1130331216755907›Hgݳ  epoch‚3 2loss/SRqqt2/assignment_loss‚0.9924125296095795 0loss/FBt2/assignment_loss‚0.1914710106329081 .loss/FBt2/detection_loss‚1.541821470213698 trainer/global_step‚2549 _step‚206 0loss/FRt1/assignment_loss‚3.1130331216755907 2loss/SRqqt1/assignment_loss‚1.1986923530628835 1loss/SRqqt1/detection_loss‚2.0010126044055343 /loss/FRt1/detection_loss‚0.5249569383377258 /loss/FRt2/detection_loss‚1.4722987138094383 0loss/SRqqt2/detection_loss‚0.733762390693189 .loss/FBt1/detection_loss‚2.496926337826057 %loss/total_loss‚16.16530656934943 0loss/FRt2/assignment_loss‚1.7875649001396878 1loss/FBt1/assignment_loss‚0.11135419894313707 $ _timestamp‚1.784502870334971e+09 _runtime‚1024.397926292Îó´'5h´ ‚(:Ý à´õÒË™è¿& disk.nvme0c0n1.out‚9498.5263671875$ memory_percent‚99.60284233093262& proc.memory.availableMB‚ 65.0703125$ disk./.usageGB‚1168.666103363037$ disk.nvme0n1.out‚9498.5263671875! disk.nvme1c1n1.in‚ 10.48046875 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 48.3671875# proc.memory.rssMB‚ 2873.37890625* proc.memory.percent‚17.537713050842285 cpu‚26.605269442520722) disk./.usagePercent‚76.12220181488598 proc.cpu.threads‚17 disk.nvme0n1.in‚ 48.3671875% disk.nvme1c1n1.out‚1338.466796875 disk.nvme1n1.in‚ 10.48046875# disk.nvme1n1.out‚1338.466796875 disk.md0.out‚ 29.203125# disk.dm-0.out‚10370.45166015625! disk.dm-1.out‚404.14990234375 disk.rbd0.out‚0 network.sent‚ 3.158763e+06 network.recv‚334261 disk.md0.in‚0 disk.dm-0.in‚ 58.84765625 disk.rbd0.in‚0AÝÞêµ ‚(:„ à´õÒê„ÂÀ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚159.372' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚53.124 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 e™2¶ - _wandbruntime‚1042  _runtime‚1042;!&ˆÉ· à trainer/global_step‚2599 % _timestamp‚1.7845028885376964e+09 _runtime‚1042.600843058 _step‚207 ' lr-AdamW/pg1‚0.00011154742417780102 ' lr-AdamW/pg2‚0.00011154742417780102\ü]θ È ' lr-AdamW/pg2‚0.00011154742417780102 trainer/global_step‚2599 % _timestamp‚1.7845028885376964e+09 _runtime‚1042.600843058 _step‚207 ' lr-AdamW/pg1‚0.00011154742417780102Ïè¡è"2¹ - _wandbruntime‚1042  _runtime‚1042Šã‰HÌ'º ÆO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19750993 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18107845 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 trainer/global_step‚2599 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19750991 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚101906243008512 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6669421 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚101906243008512 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1826296232960 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1820588045222 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46091115 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95600189092864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚100079946775552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚98782091059408 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26340122 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚97479613092864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1879212391424 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚97479401484288 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26329979 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚100079946775552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1826296232960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚100079963814912 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46080970 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11438424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚100079963814912 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1826384254976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚95600213996544 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1820674193696 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26340122 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚101906348069888 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚101906348069888 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26329979 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1826384254976 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18108105 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26329979 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1826384254976 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚18108105 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚101906243008512 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6669421 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚101906243008512 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1826296232960 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1820588045222 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46091115 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95600189092864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚100079946775552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚98782091059408 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26340122 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚97479613092864 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1879212391424 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚97479401484288 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26329979 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚100079946775552 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1826296232960 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚100079963814912 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46080970 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11438424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F@DeviceStatsMonitor.Ý‹j=on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚100079963814912 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1826384254976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0Ðt8Cý2¼ - _wandbruntime‚1042  _runtime‚10426ˆbÕ&½ ÏM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19758314 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18114522 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11442714 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚97515999020544 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚97515797904896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _step‚209 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1826974342656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚100117313142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46108348 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1826974342656 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1827062371840 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1879913654784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚100117313142784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6672066 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚100640330311780 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11442717 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26350034 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26350034 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1827062371840 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚100117330182144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1880089866752 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46098189 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚101944392553984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚101944287485440 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1821350145252 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6671808 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95635909153792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚98818963127168 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19758312 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚101944392553984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚98818980166528 % _timestamp‚1.7845028887915936e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46108348 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19758312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 trainer/global_step‚2599 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚95635884250112 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚ 1042.85487762eóÍÚ&¾ ÔM QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46098189 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚101944392553984 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚101944287485440 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1821350145252 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6671808 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95635909153792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚98818963127168 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19758312 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚101944392553984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚98818980166528 % _timestamp‚1.7845028887915936e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46108348 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19758312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 trainer/global_step‚2599 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚95635884250112 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚ 1042.85487762 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19758314 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18114522 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11442714 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚97515999020544 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚97515797904896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _step‚209 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1826974342656 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚100117313142784 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46108348 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1826974342656 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1827062371840 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1879913654784 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚100117313142784 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6672066 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚100640330311780 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11442717 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26350034 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26350034 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1827062371840 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚100117330182144 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1880089866752ÑOÚ·2¿ - _wandbruntime‚1042  _runtime‚1042Í ˜^À ý .loss/FRt1/assignment_loss‚3.22807900553888 epoch‚3 _step‚210 &loss/total_loss‚16.469642261519237 2loss/SRqqt2/assignment_loss‚1.0831767515706925 1loss/FBt1/assignment_loss‚0.21046968934408344 .loss/FRt1/detection_loss‚0.556799724450553 /loss/FRt2/detection_loss‚1.5088230959056554 1loss/SRqqt2/detection_loss‚0.7188539848567899 /loss/FBt1/detection_loss‚1.4796001863425305 $ _timestamp‚1.784502888792532e+09 2loss/SRqqt1/assignment_loss‚1.3817769021300945 1loss/FBt2/assignment_loss‚0.05669657655528226 /loss/FBt2/detection_loss‚2.7269043985203347 trainer/global_step‚2599 _runtime‚ 1042.85538553 0loss/FRt2/assignment_loss‚1.8362755429294102 1loss/SRqqt1/detection_loss‚1.6821864033749296 ¸ªÁ ‚ .loss/FRt1/assignment_loss‚3.22807900553888 epoch‚3 _step‚210 &loss/total_loss‚16.469642261519237 2loss/SRqqt2/assignment_loss‚1.0831767515706925 1loss/FBt1/assignment_loss‚0.21046968934408344 .loss/FRt1/detection_loss‚0.556799724450553 /loss/FRt2/detection_loss‚1.5088230959056554 1loss/SRqqt2/detection_loss‚0.7188539848567899 /loss/FBt1/detection_loss‚1.4796001863425305 $ _timestamp‚1.784502888792532e+09 2loss/SRqqt1/assignment_loss‚1.3817769021300945 1loss/FBt2/assignment_loss‚0.05669657655528226 /loss/FBt2/detection_loss‚2.7269043985203347 trainer/global_step‚2599 _runtime‚ 1042.85538553 0loss/FRt2/assignment_loss‚1.8362755429294102 1loss/SRqqt1/detection_loss‚1.6821864033749296Ò}Õþi ‚(:Þ ï´õÒ—Ìâ¿! disk.nvme1c1n1.in‚ 10.48046875' disk.nvme1c1n1.out‚1339.10302734375% disk.nvme1n1.out‚1339.10302734375 disk.md0.out‚29.5625 proc.cpu.threads‚17) disk./.usagePercent‚76.12219436066563 disk.dm-0.in‚ 58.84765625# disk.dm-0.out‚10371.91259765625! disk.dm-1.out‚404.21630859375 disk.nvme1n1.in‚ 10.48046875 disk.md0.in‚0$ memory_percent‚99.60308074951172* proc.memory.percent‚17.537713050842285 network.sent‚ 3.222397e+06 network.recv‚338256% disk./.usageGB‚1168.6659889221191 disk.dm-1.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚9500.18994140625$ proc.memory.availableMB‚65.03125# proc.memory.rssMB‚ 2873.37890625 disk.rbd0.out‚0 disk.nvme0n1.in‚ 48.3671875 cpu‚26.5962694253618 disk.nvme0c0n1.in‚ 48.3671875% disk.nvme0n1.out‚9500.18994140625%ŽÑà ‚(:’ ï´õÒÚŠÁÀ gpu.0.gpu‚83.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚137.165' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚45.721666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä8KÓ2Ä - _wandbruntime‚1060  _runtime‚1060¿õ—ÈÅ Â trainer/global_step‚2649 $ _timestamp‚1.784502906649552e+09 _runtime‚1060.713595436 _step‚211 ' lr-AdamW/pg1‚0.00011151480019745615 ' lr-AdamW/pg2‚0.00011151480019745615 CqÜÍÆ Ç ' lr-AdamW/pg1‚0.00011151480019745615 ' lr-AdamW/pg2‚0.00011151480019745615 trainer/global_step‚2649 $ _timestamp‚1.784502906649552e+09 _runtime‚1060.713595436 _step‚211Ó°OÒ™2Ç - _wandbruntime‚1060  _runtime‚1060hyD#È ÇO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11652927 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚102480111968038 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1860233974784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20117043 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚212 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20117043 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20117041 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚103808499111936 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6792709 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97384971858944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚103808499111936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26836874 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚101948282176512 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1860321996800 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚100625711486768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26836874 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97384946955264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚103808604173312 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚101948282176512 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚101948265137152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚103808604173312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1860233974784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18445633 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20117041 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚99299134166016 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 $ _timestamp‚1.784502906650192e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚102480215155872 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26826731 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1914365527040 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1854503669104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1860321996800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚100625694447408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46943772 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26826731 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1860233974784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20117043 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚212 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20117043 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20117041 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚103808499111936 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6792709 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚97384971858944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚103808499111936 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26836874 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚101948282176512 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1860321996800 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚100625711486768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26836874 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97384946955264 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚103808604173312 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚101948282176512 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚101948265137152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚103808604173312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1860233974784 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18445633 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0Ô2»žç2Ê - _wandbruntime‚1060  _runtime‚1060œ>¯˜Ø&Ë ÒM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6795531 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚99335523566080 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845029069030495e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26836629 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1861000113664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46960991 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11657217 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1855179620660 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46971150 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1860912084480 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18452499 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1914881453568 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1855093472074 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20124364 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚2649 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26836629 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚101985648543744 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚102517763175188 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚100662566515168 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1861000113664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚103846543588864 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20124364 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46971150 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46960991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚103846648657408 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97420642112512 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚101985631504384 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18452748 _step‚213 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚101985631504384 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20124362 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1860912084480 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚103846648657408 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6795285 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97420667016192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _runtime‚1060.966338143 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11657217 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1855179620660 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46971150 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1860912084480 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18452499 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1914881453568 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1855093472074 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20124364 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚2649 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26836629 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚101985648543744 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚102517763175188 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚100662566515168 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1861000113664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚103846543588864 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20124364 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚46971150 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46960991 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚103846648657408 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97420642112512 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚101985631504384 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18452748 _step‚213 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚101985631504384 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20124362 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1860912084480 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚103846648657408 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6795285 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97420667016192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _runtime‚1060.966338143 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845029069030495e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26836629 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1861000113664 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46960991 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105710755215360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 trainer/global_step‚2699 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20483093 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚103816600538112 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20483093 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚102469297835408 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚103816600538112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚216 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1894259738624 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20483091 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11867427 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚99169729721344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚1081.142768848 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11867424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚104357648019280 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20483091 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27323483 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1949301715456 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27333626 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6916293 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 66186117125Ûï-Ô'Ù ÎO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20483093 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚102469297835408 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚103816600538112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚216 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1894259738624 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20483091 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11867427 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚99169729721344 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚1081.142768848 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11867424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚104357648019280 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20483091 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27323483 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1949301715456 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27333626 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6916293 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6916044 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚103816583498752 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1894259738624 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27333626 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚47806574 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1949121302016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105710755215360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 trainer/global_step‚2699 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20483093 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚103816600538112 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208Ø¡M 2Ú - _wandbruntime‚1081  _runtime‚1081¿T•'#Û ÓM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚101155430377472 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚103853949865984 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6918727 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚105748904760832 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18790444 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚102506169903168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚101155212484608 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1889009096068 trainer/global_step‚2699 $ _timestamp‚1.784502927349991e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1894849826304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚99205399974912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1894937855488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚105748799692288 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚103853949865984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47823793 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1894849826304 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1888922947482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20490412 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚103853966905344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚104395196038596 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20490414 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47823793 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11871717 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20490412 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27333381 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1949812509696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18790174 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11871714 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27343538 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚105748904760832 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚102506186942528 D>DeviceStatsMonitor.on_t_K­3²rain_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47833952 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6918460 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27333381 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1950005498880 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989184 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27343538 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚99205424878592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0•¹ÃüÞ&Ü ØM E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20490412 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚103853966905344 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚104395196038596 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20490414 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚47823793 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11871717 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20490412 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27333381 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1949812509696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18790174 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11871714 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27343538 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚105748904760832 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚102506186942528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚47833952 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6918460 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27333381 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1950005498880 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989184 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27343538 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚99205424878592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚101155430377472 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚103853949865984 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6918727 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚105748904760832 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18790444 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚102506169903168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚101155212484608 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1889009096068 trainer/global_step‚2699 $ _timestamp‚1.784502927349991e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1894849826304 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚99205399974912 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1894937855488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚105748799692288 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚103853949865984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚47823793 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1894849826304 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1888922947482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482ÙZŒg:2Ý - _wandbruntime‚1081  _runtime‚1081ç‚Q]Þ û /loss/FBt2/detection_loss‚1.2320070318549243 $ _timestamp‚1.784502927351033e+09 _step‚218 .loss/FBt1/detection_loss‚4.454783483559337 /loss/FRt1/assignment_loss‚3.157498615891186 .loss/FRt2/detection_loss‚1.336484106239379 1loss/SRqqt2/detection_loss‚0.7368985288326573 trainer/global_step‚2699 _runtime‚1081.413936726 1loss/SRqqt1/detection_loss‚1.6177912411391735 %loss/total_loss‚17.09573209150809 epoch‚3 0loss/FRt2/assignment_loss‚1.5596130358255826 1loss/SRqqt1/assignment_loss‚1.280327354542911 1loss/FBt1/assignment_loss‚0.11394361007768478 1loss/FBt2/assignment_loss‚0.03227911739436747 /loss/FRt1/detection_loss‚0.6097776254861146 1loss/SRqqt2/assignment_loss‚0.964328340664771á euß € 1loss/SRqqt1/detection_loss‚1.6177912411391735 %loss/total_loss‚17.09573209150809 epoch‚3 0loss/FRt2/assignment_loss‚1.5596130358255826 1loss/SRqqt1/assignment_loss‚1.280327354542911 1loss/FBt1/assignment_loss‚0.11394361007768478 1loss/FBt2/assignment_loss‚0.03227911739436747 /loss/FRt1/detection_loss‚0.6097776254861146 1loss/SRqqt2/assignment_loss‚0.964328340664771 /loss/FBt2/detection_loss‚1.2320070318549243 $ _timestamp‚1.784502927351033e+09 _step‚218 .loss/FBt1/detection_loss‚4.454783483559337 /loss/FRt1/assignment_loss‚3.157498615891186 .loss/FRt2/detection_loss‚1.336484106239379 1loss/SRqqt2/detection_loss‚0.7368985288326573 trainer/global_step‚2699 _runtime‚1081.413936726ÚÎ*³‘à ‚(:† œµõÒ‹¬ä¿ gpu.0.gpu‚85.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚124.116' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚41.372 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûKlá ‚(:á œµõÒü‰õ¿$ disk./.usageGB‚1168.666088104248 disk.dm-1.out‚411.6611328125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 48.3671875' disk.nvme0c0n1.out‚9528.49072265625 disk.nvme1n1.in‚ 10.48046875$ memory_percent‚99.60684776306152& proc.memory.availableMB‚ 64.4140625 network.sent‚ 3.363655e+06 network.recv‚350496 disk.rbd0.in‚0 disk.nvme0n1.in‚ 48.3671875& disk.nvme1c1n1.out‚1348.2490234375 disk.md0.in‚0 disk.dm-0.in‚ 58.84765625# proc.memory.rssMB‚ 2873.37890625) disk./.usagePercent‚76.12220082098993# disk.dm-0.out‚10399.37158203125* proc.memory.percent‚17.537713050842285 cpu‚26.574773449648543 disk.dm-1.in‚0% disk.nvme0n1.out‚9528.49072265625! disk.nvme1c1n1.in‚ 10.48046875$ disk.nvme1n1.out‚1348.2490234375 disk.md0.out‚ 30.7421875 proc.cpu.threads‚17ðÏš2â - _wandbruntime‚1101  _runtime‚1101~ZKéÉã à % _timestamp‚1.7845029471341338e+09 _runtime‚1101.198574489 _step‚219 ' lr-AdamW/pg1‚0.00011144688827854893 ' lr-AdamW/pg2‚0.00011144688827854893 trainer/global_step‚2749OŸ?ÍÎä È ' lr-AdamW/pg2‚0.00011144688827854893 trainer/global_step‚2749 % _timestamp‚1.7845029471341338e+09 _runtime‚1101.198574489 _step‚219 ' lr-AdamW/pg1‚0.00011144688827854893Û ð2å - _wandbruntime‚1101  _runtime‚1101:×[Ò'æ ÌO SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7039263 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102938722320896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 _step‚220 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20849141 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚105684918899712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102938519100928 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104312901223408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48669376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27820235 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1922076471446 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27830378 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12081927 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚106234977694854 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚105684918899712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20849143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104312918262768 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1928197480448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚107613011318784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20849143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1984234737152 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚100954487583744 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1984056420864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48679521 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27830378 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚100954462680064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845029471348016e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1928197480448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12081924 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 _runtime‚1101.198631949 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚107613116380160 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7039510 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19121437 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚106235080882688 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20849141 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105684901860352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105684901860352 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚107613116380160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1928109458432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚107613011318784 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1928109458432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚107613011318784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20849143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1984234737152 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚100954487583744 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1984056420864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48679521 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27830378 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚100954462680064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845029471348016e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1928197480448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12081924 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 _runtime‚1101.198631949 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚107613116380160 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7039510 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19121437 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚106235080882688 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20849141 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚105684901860352 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105684901860352 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚107613116380160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1928109458432 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚107613011318784 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1928109458432 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚221 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1928875597312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48686595 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚107651055795712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19127867 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1928875597312 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚100990157837312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚104349773291168 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚48696754 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚107651160864256 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1922838571476 $ _timestamp‚1.784502947405163e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20856462 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27840290 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1928787568128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27830133 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102974908599296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19128114 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12086217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20856464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20856462 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7041653 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚100990182740992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚104349790330528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1928787568128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105722285266944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105722268227584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12086214 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27840290 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7041897 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚106272628902004 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2749 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48696754 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105722285266944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚48686595 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚106272525714058K2næà&ê ÚM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27830133 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1922752422890 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20856464 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚221 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1928875597312 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚48686595 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚107651055795712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19127867 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1928875597312 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚100990157837312 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚104349773291168 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚48696754 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚107651160864256 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1922838571476 $ _timestamp‚1.784502947405163e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20856462 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27840290 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1928787568128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27830133 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102974908599296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19128114 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12086217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20856464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20856462 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7041653 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚100990182740992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚104349790330528 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1928787568128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚105722285266944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚105722268227584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12086214 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27840290 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7041897 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚106272628902004 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚2749 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48696754 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105722285266944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚48686595 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚106272525714058Ýý%&2ë - _wandbruntime‚1101  _runtime‚11010B´/ì û 0loss/FRt2/assignment_loss‚1.5943295979108967 1loss/SRqqt2/assignment_loss‚1.078912202921929 0loss/FBt2/assignment_loss‚0.0895875813664927 /loss/FRt1/detection_loss‚0.5639026067302324 1loss/SRqqt2/detection_loss‚0.7602203038383482 %loss/total_loss‚18.14924435197243 _step‚222 /loss/FRt2/detection_loss‚1.3978430430317053 1loss/SRqqt1/detection_loss‚1.6247232593347154 /loss/FBt2/detection_loss‚1.2644900981213925 $ _timestamp‚1.784502947406261e+09 0loss/FRt1/assignment_loss‚3.0798297207621697 2loss/SRqqt1/assignment_loss‚1.3078568531271384 trainer/global_step‚2749 _runtime‚ 1101.46923296 0loss/FBt1/assignment_loss‚0.1475909599743041 .loss/FBt1/detection_loss‚5.239958124853106 epoch‚3\ ñí € 0loss/FRt2/assignment_loss‚1.5943295979108967 1loss/SRqqt2/assignment_loss‚1.078912202921929 0loss/FBt2/assignment_loss‚0.0895875813664927 /loss/FRt1/detection_loss‚0.5639026067302324 1loss/SRqqt2/detection_loss‚0.7602203038383482 %loss/total_loss‚18.14924435197243 _step‚222 /loss/FRt2/detection_loss‚1.3978430430317053 1loss/SRqqt1/detection_loss‚1.6247232593347154 /loss/FBt2/detection_loss‚1.2644900981213925 $ _timestamp‚1.784502947406261e+09 0loss/FRt1/assignment_loss‚3.0798297207621697 2loss/SRqqt1/assignment_loss‚1.3078568531271384 trainer/global_step‚2749 _runtime‚ 1101.46923296 0loss/FBt1/assignment_loss‚0.1475909599743041 .loss/FBt1/detection_loss‚5.239958124853106 epoch‚3ÞÍ{©K_î ‚(:Ô «µõÒ˜›ˆÀ disk.md0.out‚ 31.2890625$ memory_percent‚99.60556030273438" proc.memory.availableMB‚64.625# proc.memory.rssMB‚ 2873.37890625 cpu‚26.56598395108553 proc.cpu.threads‚17 disk.dm-0.in‚ 58.84765625" disk.dm-0.out‚10400.7529296875! disk.nvme1c1n1.in‚ 10.48046875% disk.nvme1c1n1.out‚1349.025390625 disk.nvme1n1.in‚ 10.48046875# disk.nvme1n1.out‚1349.025390625 disk.dm-1.in‚0 disk.dm-1.out‚411.7197265625 disk.nvme0c0n1.in‚ 48.3671875* proc.memory.percent‚17.537713050842285" disk.nvme0n1.out‚ 9530.31640625 network.recv‚354654) disk./.usagePercent‚76.12219659693173% disk./.usageGB‚1168.6660232543945 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 9530.31640625 disk.nvme0n1.in‚ 48.3671875 disk.md0.in‚0 network.sent‚ 3.425994e+06 disk.rbd0.out‚0áÞmœï ‚(:‘ «µõÒÇúÊ¿ gpu.0.gpu‚70.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚146.201' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚48.733666666666664 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(ƒ2ð - _wandbruntime‚1121  _runtime‚1121ãqÞîÉñ à _step‚223 ' lr-AdamW/pg1‚0.00011141160142773382 ' lr-AdamW/pg2‚0.00011141160142773382 trainer/global_step‚2799 % _timestamp‚1.7845029672236075e+09 _runtime‚1121.287610829Ÿ¹ Îò È _step‚223 ' lr-AdamW/pg1‚0.00011141160142773382 ' lr-AdamW/pg2‚0.00011141160142773382 trainer/global_step‚2799 % _timestamp‚1.7845029672236075e+09 _runtime‚1121.287610829ßa_É2ó - _wandbruntime‚1121  _runtime‚1121¹p¾½ô ÌO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚104758233290240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1962135222272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚102739220542464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚108112513746096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚109515372483584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚104758442801664 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 _step‚224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109515267422208 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚109515267422208 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2019012747776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49542323 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚106156521650768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12296424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49532178 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚107553220221952 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 trainer/global_step‚2799 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28316987 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚108112410558262 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚102739245446144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12296427 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21215193 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚107553237261312 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28316987 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1955992095328 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845029672243369e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106156504611408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7162193 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1962135222272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28327130 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19458617 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2019197355520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 _runtime‚1121.287684469 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21215191 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0Ãnà…×'õ ÑO M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚49542323 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚106156521650768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12296424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49532178 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚107553220221952 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 trainer/global_step‚2799 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28316987 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚108112410558262 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚102739245446144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12296427 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21215193 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚107553237261312 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28316987 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1955992095328 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845029672243369e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚106156504611408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7162193 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1962135222272 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28327130 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19458617 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2019197355520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 _runtime‚1121.287684469 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21215191 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚104758233290240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1962135222272 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚102739220542464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚108112513746096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚109515372483584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚104758442801664 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 _step‚224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109515267422208 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚109515267422208 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2019012747776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145àýÆ$2ö - _wandbruntime‚1121  _runtime‚1121F%º˜Ü&÷ ÖM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21222514 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚107590586589184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚109553311899136 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1962725309952 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2799 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _runtime‚1121.558907447 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2019709095936 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1956581898298 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7164955 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102774940603392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49549397 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21222514 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19465672 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚104794838494208 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106193376679168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7845029674956026e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚107590603628544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12300714 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1962725309952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚108150061765412 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106193393718528 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7164703 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28337042 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12300717 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21222512 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19465417 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49559556 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28326885 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1962813339136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49559556 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚107590603628544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚109553416967680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49549397 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21222512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28326885 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚225 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2019897890816 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚107590586589184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28337042 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚104794624795648 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1956668046884 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚28337042 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚104794624795648 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1956668046884 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2799 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _runtime‚1121.558907447 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2019709095936 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1956581898298 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7164955 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102774940603392 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49549397 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21222514 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚19465672 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚104794838494208 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚106193376679168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7845029674956026e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚107590603628544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12300714 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1962725309952 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚108150061765412 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚106193393718528 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7164703 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28337042 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12300717 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚21222512 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19465417 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49559556 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28326885 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1962813339136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49559556 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚107590603628544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚109553416967680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚49549397 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21222512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28326885 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDhÍtþeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚225 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208áj©­2ù - _wandbruntime‚1121  _runtime‚1121ò– ú ü /loss/FRt1/detection_loss‚0.6317794981027635 /loss/FRt2/detection_loss‚1.3700545998473623 /loss/FBt1/detection_loss‚4.9705831532723215 _runtime‚1121.559596178 0loss/FRt2/assignment_loss‚1.6038966368545184 2loss/SRqqt2/assignment_loss‚0.9561445393405447 1loss/SRqqt2/detection_loss‚0.6728228512993636 $ _timestamp‚1.784502967496664e+09 &loss/total_loss‚21.406262536992543 epoch‚3 -loss/FRt1/assignment_loss‚3.0445551625511 2loss/SRqqt1/assignment_loss‚1.1159664021561975 1loss/FBt1/assignment_loss‚0.17974553451674927 0loss/SRqqt1/detection_loss‚1.574573723487373 .loss/FBt2/detection_loss‚5.180640333126775 1loss/FBt2/assignment_loss‚0.10550010243747465 trainer/global_step‚2799 _step‚226T<ý1û  -loss/FRt1/assignment_loss‚3.0445551625511 2loss/SRqqt1/assignment_loss‚1.1159664021561975 1loss/FBt1/assignment_loss‚0.17974553451674927 0loss/SRqqt1/detection_loss‚1.574573723487373 .loss/FBt2/detection_loss‚5.180640333126775 1loss/FBt2/assignment_loss‚0.10550010243747465 trainer/global_step‚2799 _step‚226 /loss/FRt1/detection_loss‚0.6317794981027635 /loss/FRt2/detection_loss‚1.3700545998473623 /loss/FBt1/detection_loss‚4.9705831532723215 _runtime‚1121.559596178 0loss/FRt2/assignment_loss‚1.6038966368545184 2loss/SRqqt2/assignment_loss‚0.9561445393405447 1loss/SRqqt2/detection_loss‚0.6728228512993636 $ _timestamp‚1.784502967496664e+09 &loss/total_loss‚21.406262536992543 epoch‚3âCr ]ü ‚(:Ò ºµõÒÄ–À disk.nvme1n1.in‚ 10.48046875 disk.md0.out‚ 31.5546875" disk.dm-0.out‚10407.5107421875$ memory_percent‚99.60463047027588# proc.memory.rssMB‚ 2873.37890625 network.sent‚ 3.488415e+06 disk.dm-1.in‚0 disk.rbd0.in‚0% disk.nvme0c0n1.out‚9537.322265625# disk.nvme0n1.out‚9537.322265625* proc.memory.percent‚17.537713050842285& disk.nvme1c1n1.out‚1349.5224609375$ disk.nvme1n1.out‚1349.5224609375' proc.memory.availableMB‚ 64.77734375 proc.cpu.threads‚17 network.recv‚359099% disk./.usageGB‚1168.6660537719727 disk.dm-1.out‚ 411.8671875 disk.nvme0n1.in‚48.375! disk.nvme1c1n1.in‚ 10.48046875 disk.md0.in‚0 disk.dm-0.in‚ 58.85546875 cpu‚26.55698893489912) disk./.usagePercent‚76.12219858472383 disk.rbd0.out‚0 disk.nvme0c0n1.in‚48.375µå¬›ý ‚(: ºµõÒðæ¼À gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚199.921' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.64033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0BÓš*eþ ‚(:Ú ɵõÒêÞ‰À' proc.memory.availableMB‚ 64.77734375 network.sent‚ 3.539734e+06 network.recv‚362997$ disk.nvme1n1.out‚1350.2880859375 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚99.60463047027588# proc.memory.rssMB‚ 2873.37890625* proc.memory.percent‚17.537713050842285! disk.nvme1c1n1.in‚ 10.48046875& disk.nvme1c1n1.out‚1350.2880859375 disk.dm-1.out‚ 411.96484375 cpu‚26.548237806018495 proc.cpu.threads‚17) disk./.usagePercent‚76.12219783930179% disk./.usageGB‚1168.6660423278809! disk.nvme0n1.out‚ 9562.4296875 disk.nvme1n1.in‚ 10.48046875 disk.md0.out‚ 31.76953125 disk.dm-0.in‚ 74.5859375" disk.dm-0.out‚10432.8095703125! disk.nvme0c0n1.in‚ 64.10546875 disk.nvme0n1.in‚ 64.10546875 disk.md0.in‚0 disk.rbd0.in‚0# disk.nvme0c0n1.out‚ 9562.4296875ñk˜›ÿ ‚(: ɵõÒ³³À gpu.0.gpu‚53.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.587' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.19566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xc2€ - _wandbruntime‚1141  _runtime‚1141ÿ¥Ó¨É à % _timestamp‚1.7845029869703147e+09 _runtime‚1141.034577277 _step‚227 ' lr-AdamW/pg1‚0.00011137542806486231 ' lr-AdamW/pg2‚0.00011137542806486231 trainer/global_step‚2849ýN—q΂ È % _timestamp‚1.7845029869703147e+09 _runtime‚1141.034577277 _step‚227 ' lr-AdamW/pg1‚0.00011137542806486231 ' lr-AdamW/pg2‚0.00011137542806486231 trainer/global_step‚2849ã „)2ƒ - _wandbruntime‚1141  _runtime‚1141„ žÒ'„ ÌO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚50394980 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7285735 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1989821570736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚111417628587008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 % _timestamp‚1.7845029869710844e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12510927 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚109421538583552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚109421555622912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1995984942080 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚108000125038768 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1989735422262 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21581241 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1996072964096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚106578167357440 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _step‚228 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚109421555622912 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚50405125 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28823882 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19796659 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19796915 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7285988 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚109989843421670 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1996072964096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚111417523525632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2054164048896 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111417523525632 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28823882 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104523978404864 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚109989946609504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2053983635456 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1995984942080 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21581241 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚2849 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360cÔ¡×'… ÑO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21581243 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28813739 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1995984942080 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21581241 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚2849 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚50394980 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7285735 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1989821570736 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚111417628587008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 % _timestamp‚1.7845029869710844e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12510927 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚109421538583552 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚109421555622912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1995984942080 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚108000125038768 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1989735422262 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚21581241 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1996072964096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚106578167357440 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _step‚228 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚109421555622912 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚50405125 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28823882 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚19796659 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚19796915 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7285988 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚109989843421670 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1996072964096 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚111417523525632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2054164048896 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111417523525632 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28823882 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚104523978404864 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚109989946609504 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2053983635456 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50422358 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28823637 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111455673071104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19803263 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 trainer/global_step‚2849 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1996663051776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1990497522292 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚111455568002560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 % _timestamp‚1.7845029872405038e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21588562 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1996663051776 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12515214 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚50412199 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚108036980067168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 _step‚229 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚106614568062976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28823637 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚110027391440874 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚104559698465792 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109458904950784 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚108036997106528 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚111455673071104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2054680802304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21588564 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109458921990144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7288049 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1990411373706 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7288289 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚109458904950784ÖÙìrá&ˆ ÛM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 trainer/global_step‚2849 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1996663051776 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1990497522292 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚111455568002560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 % _timestamp‚1.7845029872405038e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21588562 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1996663051776 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12515214 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚50412199 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚108036980067168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 _step‚229 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚106614568062976 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28823637 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚110027391440874 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚104559698465792 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109458904950784 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚108036997106528 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚111455673071104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2054680802304 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21588564 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚109458921990144 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7288049 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1990411373706 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7288289 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚109458904950784 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚50422358 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28823637 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111455673071104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19803263åu=¢M2‰ - _wandbruntime‚1141  _runtime‚1141 /çæŠ € 0loss/FRt1/assignment_loss‚3.1394996290218153 0loss/FRt2/assignment_loss‚1.5737237841482983 2loss/SRqqt2/assignment_loss‚1.1254124177112268 /loss/FRt2/detection_loss‚1.3861832566950965 _step‚230 /loss/FBt1/assignment_loss‚0.172192708423223 1loss/FBt2/assignment_loss‚0.08831835603717247 /loss/FRt1/detection_loss‚0.5961228477854265 &loss/total_loss‚15.144198080245697 2loss/SRqqt1/assignment_loss‚1.3306554507856305 1loss/SRqqt1/detection_loss‚1.4532546713203767 /loss/FBt1/detection_loss‚2.0960366009384632 /loss/FBt2/detection_loss‚1.3697930659764097 epoch‚3 trainer/global_step‚2849 _runtime‚1141.304440943 1loss/SRqqt2/detection_loss‚0.8130052914025592 % _timestamp‚1.7845029872415252e+09xƒZ‘ ‹ … 0loss/FRt1/assignment_loss‚3.1394996290218153 0loss/FRt2/assignment_loss‚1.5737237841482983 2loss/SRqqt2/assignment_loss‚1.1254124177112268 /loss/FRt2/detection_loss‚1.3861832566950965 _step‚230 /loss/FBt1/assignment_loss‚0.172192708423223 1loss/FBt2/assignment_loss‚0.08831835603717247 /loss/FRt1/detection_loss‚0.5961228477854265 &loss/total_loss‚15.144198080245697 2loss/SRqqt1/assignment_loss‚1.3306554507856305 1loss/SRqqt1/detection_loss‚1.4532546713203767 /loss/FBt1/detection_loss‚2.0960366009384632 /loss/FBt2/detection_loss‚1.3697930659764097 epoch‚3 trainer/global_step‚2849 _runtime‚1141.304440943 1loss/SRqqt2/detection_loss‚0.8130052914025592 % _timestamp‚1.7845029872415252e+09æ:j;øaŒ ‚(:Ö صõÒ¯æÀ% disk.nvme1n1.out‚1350.93115234375 disk.dm-0.in‚ 90.88671875 disk.dm-1.in‚0 disk.nvme0c0n1.in‚80.40625# disk.nvme0c0n1.out‚ 9568.6484375! disk.nvme1c1n1.in‚ 10.48046875* proc.memory.percent‚17.537713050842285 network.recv‚366676 disk.nvme1n1.in‚ 10.48046875 disk.md0.out‚32.1875! disk.dm-1.out‚412.06103515625! disk.nvme0n1.out‚ 9568.6484375' disk.nvme1c1n1.out‚1350.93115234375 proc.cpu.threads‚17% disk./.usageGB‚1168.6661224365234 disk.md0.in‚0 disk.rbd0.in‚0$ memory_percent‚99.60486888885498' proc.memory.availableMB‚ 64.73828125# proc.memory.rssMB‚ 2873.37890625 network.sent‚ 3.555547e+06) disk./.usagePercent‚76.12220305725604" disk.dm-0.out‚10438.6923828125 disk.rbd0.out‚0 disk.nvme0n1.in‚80.40625 cpu‚26.540132747609725|·Ì‘ ‚(:† صõÒ¼¶À gpu.0.gpu‚67.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚176.958' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚58.986 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qŸ:2Ž - _wandbruntime‚1161  _runtime‚1161Ð! ÎÉ Ã _runtime‚1161.137319881 _step‚231 ' lr-AdamW/pg1‚0.00011133836876932421 ' lr-AdamW/pg2‚0.00011133836876932421 trainer/global_step‚2899 % _timestamp‚1.7845030070741384e+09Ö¡Î È % _timestamp‚1.7845030070741384e+09 _runtime‚1161.137319881 _step‚231 ' lr-AdamW/pg1‚0.00011133836876932421 ' lr-AdamW/pg2‚0.00011133836876932421 trainer/global_step‚2899çéï¢2‘ - _wandbruntime‚1161  _runtime‚1161ž.jÒ'’ ÌO @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚109843728426768 % _timestamp‚1.7845030070753238e+09 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51267927 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12725424 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚106308736267264 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚111289873984512 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20134661 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111867379472912 _step‚232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113319884690432 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7409486 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108397719230464 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚111289856945152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21947291 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29320634 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12725427 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2030010705920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7409237 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2088982963200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51257782 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113319779629056 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113319884690432 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚106308761170944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 trainer/global_step‚2899 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚111289856945152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2029922683904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29320634 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108397924547584 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚109843711387408 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2023564897670 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2029922683904 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚113319779629056 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2030010705920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29310491 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2023651046144 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29310491 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21947293 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111289873984512 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚1161.138565303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21947293 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2089163376640 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2088982963200 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚51257782 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113319779629056 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚113319884690432 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚106308761170944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 trainer/global_step‚2899 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚111289856945152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2029922683904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29320634 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108397924547584 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚109843711387408 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2023564897670 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2029922683904 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚113319779629056 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2030010705920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29310491 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2023651046144 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29310491 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21947293 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111289873984512 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚1161.138565303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21947293 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2089163376640 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚111289873984512 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20134661 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111867379472912 _step‚232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113319884690432 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7409486 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108397719230464 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚111289856945152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21947291 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29320634 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12725427 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2030010705920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7409237èuv£}2” - _wandbruntime‚1161  _runtime‚1161Bò Ü&• ÖM E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚106344456328192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚109880583455168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113357929174528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108434320317952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51275001 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2030600793600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚109880600494528 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7411921 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113357929174528 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51285160 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20141389 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111904824304282 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 _step‚233 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 % _timestamp‚1.7845030073483963e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21954612 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12729717 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 trainer/global_step‚2899 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7411675 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21954614 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚111327223312384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111904927492228 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Ñ=þ÷á&– ÛM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12729717 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 trainer/global_step‚2899 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7411675 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21954614 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚111327223312384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111904927492228 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚106344456328192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚109880583455168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚113357929174528 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚108434320317952 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚51275001 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2030600793600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚109880600494528 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7411921 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚113357929174528 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51285160 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20141389 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111904824304282 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 _step‚233 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 % _timestamp‚1.7845030073483963e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21954612 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0é:72— - _wandbruntime‚1161  _runtime‚1161™ZˆË˜ ü .loss/FRt2/detection_loss‚1.390097126188373 % _timestamp‚1.7845030073495822e+09 epoch‚3 /loss/FRt2/assignment_loss‚1.654774183736128 1loss/SRqqt1/assignment_loss‚1.095687482469657 1loss/FBt1/assignment_loss‚0.24616753512709466 1loss/SRqqt1/detection_loss‚1.6721211031995722 1loss/SRqqt2/detection_loss‚0.8210986815791008 /loss/FBt1/detection_loss‚2.7649652985298783 .loss/FBt2/detection_loss‚1.211217863364673 2loss/SRqqt2/assignment_loss‚0.9533870893127141 0loss/FBt2/assignment_loss‚0.0900873167467509 /loss/FRt1/detection_loss‚0.6265380199910264 trainer/global_step‚2899 _step‚234 &loss/total_loss‚15.636662155991175 0loss/FRt1/assignment_loss‚3.1105204557462067 _runtime‚ 1161.41248227ƒ&™  epoch‚3 /loss/FRt2/assignment_loss‚1.654774183736128 1loss/SRqqt1/assignment_loss‚1.095687482469657 1loss/FBt1/assignment_loss‚0.24616753512709466 1loss/SRqqt1/detection_loss‚1.6721211031995722 1loss/SRqqt2/detection_loss‚0.8210986815791008 /loss/FBt1/detection_loss‚2.7649652985298783 .loss/FBt2/detection_loss‚1.211217863364673 2loss/SRqqt2/assignment_loss‚0.9533870893127141 0loss/FBt2/assignment_loss‚0.0900873167467509 /loss/FRt1/detection_loss‚0.6265380199910264 trainer/global_step‚2899 _step‚234 &loss/total_loss‚15.636662155991175 0loss/FRt1/assignment_loss‚3.1105204557462067 _runtime‚ 1161.41248227 .loss/FRt2/detection_loss‚1.390097126188373 % _timestamp‚1.7845030073495822e+09êÊ3}fš ‚(:Û çµõÒ»¬ˆÀ% disk./.usageGB‚1168.6660957336426 disk.rbd0.in‚0 disk.nvme0n1.in‚80.40625! disk.nvme1c1n1.in‚ 10.48046875) disk./.usagePercent‚76.12220131793796 disk.rbd0.out‚0& disk.nvme0c0n1.out‚9575.3056640625 disk.dm-0.in‚ 90.88671875! disk.dm-1.out‚412.13134765625% proc.memory.availableMB‚ 64.703125* proc.memory.percent‚17.537713050842285 cpu‚26.532025338642782 network.sent‚ 3.618037e+06 network.recv‚370715 disk.nvme0c0n1.in‚80.40625' disk.nvme1c1n1.out‚1351.81298828125% disk.nvme1n1.out‚1351.81298828125 disk.dm-1.in‚0 proc.cpu.threads‚17$ disk.nvme0n1.out‚9575.3056640625 disk.nvme1n1.in‚ 10.48046875 disk.md0.in‚0 disk.md0.out‚32.59375" disk.dm-0.out‚10445.2802734375$ memory_percent‚99.60508346557617# proc.memory.rssMB‚ 2873.37890625(ôà › ‚(:… çµõҠȱÀ gpu.0.gpu‚64.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚122.643' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.881 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U:‘2œ - _wandbruntime‚1181  _runtime‚1181¾záÇ Á _step‚235 & lr-AdamW/pg1‚0.0001113004241346994 & lr-AdamW/pg2‚0.0001113004241346994 trainer/global_step‚2949 % _timestamp‚1.7845030271229708e+09 _runtime‚1181.186137081±((Ìž Æ _step‚235 & lr-AdamW/pg1‚0.0001113004241346994 & lr-AdamW/pg2‚0.0001113004241346994 trainer/global_step‚2949 % _timestamp‚1.7845030271229708e+09 _runtime‚1181.186137081ë!M‘Û2Ÿ - _wandbruntime‚1181  _runtime‚1181ÁÇZ‡  ÈO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 trainer/global_step‚2949 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚113158192346112 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚113158175306752 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12939927 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115222035732480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚52120584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113744709148486 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29817386 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2063860425728 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2057394373078 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚113158192346112 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚52130729 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2057480521552 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108093494129664 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚115222035732480 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚113158175306752 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2063948447744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20473016 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22313343 [IDeviceStatsMonitor.on_train_batc;,8Ö¯h_start/active_bytes.small_pool.allocated‚ 2063948447744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2063860425728 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29807243 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115222140793856 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29817386 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚113744812336320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20473278 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚115222140793856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚111687314775408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22313341 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12939924 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7533351 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2124144865792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚110217663899136 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29807243 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2123960258048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 _step‚236 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22313341 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ! _timestamp‚1.784503027124e+09 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 _runtime‚1181.187171863o¤ÂSÓ'¡ ÍO _runtime‚1181.187171863 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 trainer/global_step‚2949 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚113158192346112 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚113158175306752 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12939927 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115222035732480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚52120584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚113744709148486 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29817386 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2063860425728 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2057394373078 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚113158192346112 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚52130729 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2057480521552 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚108093494129664 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚115222035732480 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚113158175306752 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2063948447744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20473016 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22313343 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2063948447744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2063860425728 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29807243 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115222140793856 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29817386 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚113744812336320 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20473278 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚115222140793856 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚111687314775408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚22313341 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12939924 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7533351 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2124144865792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚110217663899136 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚29807243 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2123960258048 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 _step‚236 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22313341 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ! _timestamp‚1.784503027124e+09 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208ìeÛl×2¢ - _wandbruntime‚1181  _runtime‚1181ĘÂÜ&£ ÖM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22320662 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2064538535424 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2064538535424 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2058156473108 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29827298 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2949 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7535725 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108129214190592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12944214 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚113195558713344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29817141 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2064626564608 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20479942 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2124663655424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚52137803 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111724203882528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52147962 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20479687 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22320664 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2124837770240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0”: uM¤ ÛM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22320662 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2064538535424 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2064538535424 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2058156473108 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29827298 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2949 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7535725 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚108129214190592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12944214 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚113195558713344 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚29817141 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2064626564608 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20479942 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2124663655424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚52137803 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111724203882528 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚52147962 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20479687 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22320664 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2124837770240 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482ížÍ2¥ - _wandbruntime‚1181  _runtime‚11814dLr¦ € /loss/FBt1/detection_loss‚1.7677098806749156 epoch‚3 _step‚238 /loss/FBt2/detection_loss‚1.8640317856257016 &loss/total_loss‚15.302167756631512 0loss/FRt2/assignment_loss‚1.4238082163984125 2loss/SRqqt1/assignment_loss‚1.3023669096951684 1loss/FBt1/assignment_loss‚0.34912690156008486 1loss/SRqqt1/detection_loss‚1.5708737818834682 1loss/SRqqt2/detection_loss‚0.7353548205726819 trainer/global_step‚2949 /loss/FRt1/assignment_loss‚3.158102254889327 2loss/SRqqt2/assignment_loss‚0.9443220036767981 _runtime‚1181.459780744 0loss/FBt2/assignment_loss‚0.2350491872577755 /loss/FRt1/detection_loss‚0.5268974697850454 /loss/FRt2/detection_loss‚1.4245245446121342 % _timestamp‚1.7845030273967397e+09°öŸ § … 1loss/SRqqt2/detection_loss‚0.7353548205726819 trainer/global_step‚2949 /loss/FRt1/assignment_loss‚3.158102254889327 2loss/SRqqt2/assignment_loss‚0.9443220036767981 _runtime‚1181.459780744 0loss/FBt2/assignment_loss‚0.2350491872577755 /loss/FRt1/detection_loss‚0.5268974697850454 /loss/FRt2/detection_loss‚1.4245245446121342 % _timestamp‚1.7845030273967397e+09 /loss/FBt1/detection_loss‚1.7677098806749156 epoch‚3 _step‚238 /loss/FBt2/detection_loss‚1.8640317856257016 &loss/total_loss‚15.302167756631512 0loss/FRt2/assignment_loss‚1.4238082163984125 2loss/SRqqt1/assignment_loss‚1.3023669096951684 1loss/FBt1/assignment_loss‚0.34912690156008486 1loss/SRqqt1/detection_loss‚1.5708737818834682îó!Äñc¨ ‚(:Ø öµõÒÚªô¿ disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚9580.45068359375 disk.nvme0n1.in‚80.40625 disk.nvme1n1.in‚ 10.48046875# proc.memory.rssMB‚ 2873.37890625* proc.memory.percent‚17.537713050842285 disk.dm-0.in‚ 90.88671875 disk.dm-1.in‚0% disk.nvme0n1.out‚9580.45068359375! disk.nvme1c1n1.in‚ 10.48046875# disk.nvme1n1.out‚1352.400390625 disk.md0.out‚ 33.00390625% proc.memory.availableMB‚ 64.640625 cpu‚26.5239175694728$ memory_percent‚99.60546493530273 proc.cpu.threads‚17 network.sent‚ 3.667501e+06 network.recv‚374964) disk./.usagePercent‚75.94526024094584$ disk./.usageGB‚1165.949607849121 disk.dm-1.out‚ 412.216796875 disk.nvme0c0n1.in‚80.40625% disk.nvme1c1n1.out‚1352.400390625 disk.md0.in‚0# disk.dm-0.out‚10450.05615234375´8Ù›© ‚(: öµõÒˆØÈÀ gpu.0.gpu‚55.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚200.093' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.69766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07ý`ª ‚(:Õ …¶õÒµü‡À disk.dm-1.in‚0 disk.rbd0.in‚0$ disk.nvme0n1.out‚9595.6005859375 disk.md0.in‚0 disk.md0.out‚33.3125# proc.memory.rssMB‚ 2873.37890625* proc.memory.percent‚17.537713050842285 cpu‚26.515995959748434! disk.nvme1c1n1.in‚ 10.48046875$ disk.nvme1n1.out‚1354.7001953125 proc.cpu.threads‚17 network.recv‚379166) disk./.usagePercent‚75.94527341006847% disk./.usageGB‚1165.9498100280762 disk.nvme0c0n1.in‚80.40625 disk.nvme1n1.in‚ 10.48046875$ memory_percent‚99.60551261901855& proc.memory.availableMB‚ 64.6328125 network.sent‚ 3.731906e+06 disk.dm-0.in‚ 90.88671875 disk.dm-0.out‚10466.69140625 disk.dm-1.out‚ 412.345703125 disk.rbd0.out‚0& disk.nvme0c0n1.out‚9595.6005859375 disk.nvme0n1.in‚80.40625& disk.nvme1c1n1.out‚1354.7001953125¥÷ }« ‚(:„ …¶õÒ—›¶À gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚198.519' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚66.173 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ôȬ52¬ - _wandbruntime‚1201  _runtime‚1201grVóÉ­ à trainer/global_step‚2999 % _timestamp‚1.7845030472133377e+09 _runtime‚1201.277145931 _step‚239 ' lr-AdamW/pg1‚0.00011126159476874817 ' lr-AdamW/pg2‚0.00011126159476874817×xή È % _timestamp‚1.7845030472133377e+09 _runtime‚1201.277145931 _step‚239 ' lr-AdamW/pg1‚0.00011126159476874817 ' lr-AdamW/pg2‚0.00011126159476874817 trainer/global_step‚299952¯ - _wandbruntime‚1201  _runtime‚1201SÅÈÒ'° ÌO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2097798167552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚109878276895744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845030472141147e+09 _runtime‚1201.277481032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13154427 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13154424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7657193 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219997184 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2158956838912 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2159151932416 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7656930 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2097886189568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30303995 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚112037208830976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22679393 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚113530935202768 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚117124396897280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚112037428828160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚115622245199728 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2097798167552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20811354 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚115026510707712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 195093504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚113530918163408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2999 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚117124291835904 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2097886189568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22679391 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚115026493668352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚52993531 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482%oîê×'± ÑO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13154424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚7657193 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219997184 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2158956838912 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2159151932416 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7656930 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2097886189568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30303995 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚112037208830976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚22679393 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚113530935202768 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚117124396897280 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚112037428828160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚115622245199728 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2097798167552 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20811354 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚115026510707712 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 195093504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚113530918163408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚2999 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚117124291835904 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2097886189568 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22679391 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚115026493668352 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚52993531 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2097798167552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚109878276895744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845030472141147e+09 _runtime‚1201.277481032 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13154427ð—Æ™2² - _wandbruntime‚1201  _runtime‚1201ö…´ºö³ ÕM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30313893 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115063877074944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2098564306432 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2098564306432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2098476277248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚117162441381376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53000605 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13158717 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2159658270208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30313893 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 trainer/global_step‚2999 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53000605 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7659379 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚112073605419520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚113567790231168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22686714 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚115659793219044 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30324050 REDeviceStatsMonitor.on_train_batch_end/actùÑñ(åive_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53010764 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30324050 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 $ _timestamp‚1.784503047485278e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2098476277248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚117162336312832 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 _step‚241 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚115063860035584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22686714 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚109913947149312 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚115063860035584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2091985948516 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20818093 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚112073812826624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚115063877074944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22686712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20818340 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7659623 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚117162441381376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482$ƒQà&´ ÚM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13158717 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2159658270208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30313893 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 trainer/global_step‚2999 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53000605 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7659379 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚112073605419520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚113567790231168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22686714 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚115659793219044 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30324050 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53010764 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30324050 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 $ _timestamp‚1.784503047485278e+09 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2098476277248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚117162336312832 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 _step‚241 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚115063860035584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22686714 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚109913947149312 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚115063860035584 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2091985948516 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20818093 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚112073812826624 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚115063877074944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22686712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚20818340 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7659623 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚117162441381376 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚30313893 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115063877074944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2098564306432 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2098564306432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2098476277248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚117162441381376 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53000605 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330ñ„V¬ƒ2µ - _wandbruntime‚1201  _runtime‚1201«Ã¶ ý % _timestamp‚1.7845030474862437e+09 _runtime‚1201.549153449 _step‚242 /loss/FRt1/detection_loss‚0.6282059892509037 epoch‚3 2loss/SRqqt2/assignment_loss‚0.9171509845035681 1loss/SRqqt2/detection_loss‚0.7468764161159137 /loss/FBt1/detection_loss‚1.8787343078455345 -loss/FBt2/detection_loss‚1.44960060995814 /loss/FRt1/assignment_loss‚2.906803539243795 /loss/FRt2/assignment_loss‚1.661210969995005 2loss/SRqqt1/assignment_loss‚1.1474808002100654 .loss/FRt2/detection_loss‚1.401360437157192 trainer/global_step‚2999 1loss/FBt1/assignment_loss‚0.17118135213086874 1loss/FBt2/assignment_loss‚0.07387317012331435 1loss/SRqqt1/detection_loss‚1.4780307196096287 &loss/total_loss‚14.460509296143929่· ‚ /loss/FRt2/assignment_loss‚1.661210969995005 2loss/SRqqt1/assignment_loss‚1.1474808002100654 .loss/FRt2/detection_loss‚1.401360437157192 trainer/global_step‚2999 1loss/FBt1/assignment_loss‚0.17118135213086874 1loss/FBt2/assignment_loss‚0.07387317012331435 1loss/SRqqt1/detection_loss‚1.4780307196096287 &loss/total_loss‚14.460509296143929 % _timestamp‚1.7845030474862437e+09 _runtime‚1201.549153449 _step‚242 /loss/FRt1/detection_loss‚0.6282059892509037 epoch‚3 2loss/SRqqt2/assignment_loss‚0.9171509845035681 1loss/SRqqt2/detection_loss‚0.7468764161159137 /loss/FBt1/detection_loss‚1.8787343078455345 -loss/FBt2/detection_loss‚1.44960060995814 /loss/FRt1/assignment_loss‚2.906803539243795ò~?‰)d¸ ‚(:Ù ”¶õÒä÷ø¿ network.recv‚382530 disk.rbd0.in‚0 disk.nvme0n1.in‚80.40625 disk.md0.in‚0* proc.memory.percent‚17.537713050842285% disk.nvme0c0n1.out‚9613.298828125% disk.nvme1n1.out‚1355.50244140625# proc.memory.rssMB‚ 2873.37890625 cpu‚26.50827999742922 proc.cpu.threads‚17 network.sent‚ 3.734929e+06 disk.nvme0c0n1.in‚80.40625! disk.nvme1c1n1.in‚ 10.48046875' disk.nvme1c1n1.out‚1355.50244140625 disk.nvme1n1.in‚ 10.48046875 disk.dm-0.in‚ 90.88671875 disk.dm-1.in‚0$ memory_percent‚99.60393905639648) disk./.usagePercent‚75.94524905961532% disk./.usageGB‚1165.9494361877441 disk.rbd0.out‚0# disk.nvme0n1.out‚9613.298828125 disk.md0.out‚ 33.85546875 disk.dm-0.out‚10483.91796875! disk.dm-1.out‚412.47314453125% proc.memory.availableMB‚ 64.890625„þÁ›¹ ‚(: ”¶õÒßÓœÀ gpu.0.gpu‚71.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚192.021' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚64.00699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž™Óî2º - _wandbruntime‚1221  _runtime‚1221&;_É» à _runtime‚1221.349161487 _step‚243 ' lr-AdamW/pg1‚0.00011122188129340168 ' lr-AdamW/pg2‚0.00011122188129340168 trainer/global_step‚3049 % _timestamp‚1.7845030672860255e+09_ Þμ È _step‚243 ' lr-AdamW/pg1‚0.00011122188129340168 ' lr-AdamW/pg2‚0.00011122188129340168 trainer/global_step‚3049 % _timestamp‚1.7845030672860255e+09 _runtime‚1221.349161487óeX¹ô2½ - _wandbruntime‚1221  _runtime‚1221ÛîÒ'¾ ÌO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚115374521551408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30800747 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚116894829069312 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2131735909376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30810890 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13368927 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚116894812029952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23045443 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2125139472368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚117499574875302 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 _runtime‚1221.353645604 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2193971656192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚115374538590768 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚119026653000704 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53846188 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2131823931392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119026547939328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7780760 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13368924 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 % _timestamp‚1.7845030672869885e+09 _step‚244 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23045443 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30810890 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚116894812029952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚117499678063136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2194158361088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751‰Á׿ ÑO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 % _timestamp‚1.7845030672869885e+09 _step‚244 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23045443 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30810890 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚116894812029952 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚117499678063136 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2194158361088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚115374521551408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30800747 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚116894829069312 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2131735909376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30810890 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13368927 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚116894812029952 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23045443 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2125139472368 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚117499574875302 G?DeviceStatsMonitor.on‘[œZ_train_batch_start/inactive_split.all.peak‚717 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 _runtime‚1221.353645604 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2193971656192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚115374538590768 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚119026653000704 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53846188 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2131823931392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119026547939328 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7780760 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13368924 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104ôµ(÷2À - _wandbruntime‚1221  _runtime‚1221ÇCLbÛ&Á ÕM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23052762 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53873566 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚117537226082452 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30820802 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2132502048256 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21156708 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚113893589731328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚115411410658528 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53873566 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119064592416256 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚117537122894506 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2125815423924 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13373214 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7783491 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2194668923904 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 trainer/global_step‚3049 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2125729275338 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚116932178397184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚116932178397184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23052764 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚116932195436544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚116932195436544 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 _runtime‚1221.620691025 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53863407 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚119064697484800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2132502048256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21156448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚115411393619168 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53873566 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚117537226082452 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30820802 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2132502048256 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21156708 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚113893589731328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚115411410658528 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53873566 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119064592416256 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚117537122894506 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2125815423924 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13373214 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7783491 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2194668923904 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 trainer/global_step‚3049 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2125729275338 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚116932178397184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚116932178397184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23052764 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚116932195436544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚116932195436544 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 _runtime‚1221.620691025 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53863407 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚119064697484800 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2132502048256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21156448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚115411393619168 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚113447792620544 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31297499 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2158882799302 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2229155278336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117218124939408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚115676947898880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2158968947776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31307642 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 F@DeviceStat—A*écsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 trainer/global_step‚3099 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚120928909104128 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7904141 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54719135 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚117218141978768 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23411491 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120928804042752 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2165761673216 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31307642 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2228968573440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚120928804042752 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚118763147430912 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚118763130391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13583427 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 % _timestamp‚1.7845030870495517e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚120928909104128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13583424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2165673651200 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21487830 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚113447767716864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119377007738710 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 _step‚248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23411493 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚113447792620544 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31297499 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2158882799302 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2229155278336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117218124939408 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚115676947898880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2158968947776 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31307642 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 trainer/global_step‚3099 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚120928909104128 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7904141 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54719135 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚117218141978768 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23411491 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚120928804042752 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2165761673216 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31307642 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2228968573440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚120928804042752 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚118763147430912 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚118763130391552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13583427 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360ø ”šc2Î - _wandbruntime‚1241  _runtime‚1241Qd¶ËÛ&Ï ÕM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54736368 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31307397 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚113483487777792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2166351760896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚115713336198144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54736368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 _step‚249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23418814 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119414555757914 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117255014046528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2159644899332 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23418812 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31307397 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2166439790080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7906612 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13587714 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2159558750746 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115713128791040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117254997007168 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2166439790080 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21494326 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚1241.381219076 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚118800513798144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚118800496758784 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2166351760896 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 $ _timestamp‚1.784503087317923e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚120966848519680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚118800496758784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13587717 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482Ý^Œs–Ð ÚM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54736368 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚31307397 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚113483487777792 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2166351760896 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚115713336198144 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54736368 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 _step‚249 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23418814 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119414555757914 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117255014046528 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2159644899332 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23418812 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31307397 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2166439790080 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7906612 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13587714 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2159558750746 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚115713128791040 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117254997007168 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2166439790080 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21494326 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚1241.381219076 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚118800513798144 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚118800496758784 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2166351760896 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 $ _timestamp‚1.784503087317923e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚120966848519680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚118800496758784 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13587717 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152ù8ä¨@2Ñ - _wandbruntime‚1241  _runtime‚1241VÂÒ ý trainer/global_step‚3099 % _timestamp‚1.7845030873189943e+09 _runtime‚1241.381948557 0loss/FRt2/assignment_loss‚1.4634800891892439 0loss/FRt1/assignment_loss‚2.8941595124484207 2loss/SRqqt2/assignment_loss‚0.9407682659733049 1loss/SRqqt2/detection_loss‚0.6900596622060076 /loss/FBt1/detection_loss‚0.9750030799730772 epoch‚3 _step‚250 0loss/FBt2/assignment_loss‚0.2941772114891137 /loss/FRt1/detection_loss‚0.5578963873034273 .loss/FRt2/detection_loss‚1.348922293304567 /loss/FBt2/detection_loss‚1.9647549602194574 &loss/total_loss‚14.484550781551704 1loss/SRqqt1/assignment_loss‚1.239578313799575 0loss/FBt1/assignment_loss‚0.2955968776779661 0loss/SRqqt1/detection_loss‚1.820154127967544%Ÿ2òÓ ‚ 0loss/FRt1/assignment_loss‚2.8941595124484207 2loss/SRqqt2/assignment_loss‚0.9407682659733049 1loss/SRqqt2/detection_loss‚0.6900596622060076 /loss/FBt1/detection_loss‚0.9750030799730772 epoch‚3 _step‚250 0loss/FBt2/assignment_loss‚0.2941772114891137 /loss/FRt1/detection_loss‚0.5578963873034273 .loss/FRt2/detection_loss‚1.348922293304567 /loss/FBt2/detection_loss‚1.9647549602194574 &loss/total_loss‚14.484550781551704 1loss/SRqqt1/assignment_loss‚1.239578313799575 0loss/FBt1/assignment_loss‚0.2955968776779661 0loss/SRqqt1/detection_loss‚1.820154127967544 trainer/global_step‚3099 % _timestamp‚1.7845030873189943e+09 _runtime‚1241.381948557 0loss/FRt2/assignment_loss‚1.4634800891892439úÀ€¬§bÔ ‚(:× ²¶õÒÔ­†À disk.rbd0.in‚0 cpu‚26.493384501746917 proc.cpu.threads‚17 network.sent‚ 3.872353e+06 network.recv‚390427! disk.nvme1c1n1.in‚ 10.48046875" disk.nvme1n1.out‚ 1891.94140625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚86.28125$ memory_percent‚99.60606098175049# proc.memory.rssMB‚ 2873.37890625 disk.nvme1n1.in‚ 10.48046875 disk.md0.in‚0 disk.md0.out‚34.4375 disk.rbd0.out‚0& disk.nvme0c0n1.out‚9624.8642578125 disk.nvme0n1.in‚86.28125$ disk.nvme0n1.out‚9624.8642578125' proc.memory.availableMB‚ 64.54296875! disk.dm-1.out‚947.35693359375% disk./.usageGB‚1165.9498138427734$ disk.nvme1c1n1.out‚ 1891.94140625# disk.dm-0.out‚10495.76416015625* proc.memory.percent‚17.537713050842285) disk./.usagePercent‚75.94527365854248 disk.dm-0.in‚ 96.76171875>I;<Õ ‚(:… ²¶õҿíÀ gpu.0.gpu‚54.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚195.798' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚65.266 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zƒªAiÖ ‚(:Þ Á¶õÒƒ¦ç¿ network.recv‚394117) disk./.usagePercent‚76.01053809094674 disk.rbd0.out‚0 disk.nvme1n1.in‚ 10.48046875' proc.memory.availableMB‚ 64.78515625# proc.memory.rssMB‚ 2873.37890625 cpu‚26.48621275748387 network.sent‚ 3.921155e+06% disk./.usageGB‚1166.9517860412598! disk.dm-0.out‚10540.287109375 disk.dm-1.in‚0! disk.dm-1.out‚947.45068359375& disk.nvme0c0n1.out‚9669.1181640625$ disk.nvme0n1.out‚9669.1181640625' disk.nvme1c1n1.out‚1892.81298828125 disk.nvme0c0n1.in‚86.28125! disk.nvme1c1n1.in‚ 10.48046875% disk.nvme1n1.out‚1892.81298828125 disk.md0.in‚0 disk.md0.out‚ 34.66796875$ memory_percent‚99.60458278656006* proc.memory.percent‚17.537713050842285 proc.cpu.threads‚17 disk.dm-0.in‚ 96.76171875 disk.rbd0.in‚0 disk.nvme0n1.in‚86.28125cÝ÷Ç›× ‚(: Á¶õÒýŽ¦À gpu.0.gpu‚78.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚118.525' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.50833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊÅå¸2Ø - _wandbruntime‚1261  _runtime‚1261õ÷&nÆÙ À $ _timestamp‚1.784503107116989e+09 _runtime‚1261.180547352 _step‚251 & lr-AdamW/pg1‚0.0001111398045730411 & lr-AdamW/pg2‚0.0001111398045730411 trainer/global_step‚3149¤ç>©ËÚ Å _step‚251 & lr-AdamW/pg1‚0.0001111398045730411 & lr-AdamW/pg2‚0.0001111398045730411 trainer/global_step‚3149 $ _timestamp‚1.784503107116989e+09 _runtime‚1261.180547352û¾²|ò2Û - _wandbruntime‚1261  _runtime‚1261ÑSÑ'Ü ËO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2199611393024 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2199699415040 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2199611393024 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚121254440602118 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 trainer/global_step‚3149 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23777541 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21825843 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚122831060146176 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8027919 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55571792 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚119061745366768 $ _timestamp‚1.784503107117708e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23777543 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2263940059648 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 _step‚252 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31804394 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13797924 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2192712274710 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 _runtime‚1261.180750672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23777541 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199025664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31804394 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚120631448753152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23777543 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117496465638912 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚120631465792512 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23777541 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21825843 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚122831060146176 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8027919 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚55571792 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚119061745366768 $ _timestamp‚1.784503107117708e+09 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚23777543 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2263940059648 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 _step‚252 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31804394 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13797924 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2192712274710 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 _runtime‚1261.180750672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23777541 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199025664 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31804394 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚120631448753152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23777543 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚117496465638912 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚120631465792512 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2193388226154 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55589011 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21832849 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55599170 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55589011 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55599170 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23784864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚120668832159744 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚121292091809268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117533065460736 trainer/global_step‚3149 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31814306 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.smalö˜F—l_pool.allocated‚ 2264819820544 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2200377531904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2200289502720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31814306 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13802217 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117532851762176 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21832600 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚122869104623104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31804149 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 % _timestamp‚1.7845031073887444e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚120668832159744 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23784864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2264631025664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚120668815120384 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8030632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚119098617434528 _step‚253 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2200289502720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚122869209691648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104¤"Qá&à ÛM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚115268245640192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚115268220736512 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚31814306 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13802217 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117532851762176 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21832600 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚122869104623104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31804149 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 % _timestamp‚1.7845031073887444e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚120668832159744 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚23784864 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2264631025664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚120668815120384 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8030632 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚119098617434528 _step‚253 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2200289502720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚122869209691648 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚23784862 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚122869209691648 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2193388226154 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚55589011 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21832849 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚55599170 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚55589011 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚55599170 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23784864 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚120668832159744 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚121292091809268 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117533065460736 trainer/global_step‚3149 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚31814306 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2264819820544 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2200377531904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2200289502720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚124733316249600 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8150815 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚124733316249600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14012424 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚124733421310976 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8150558 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2226541750118 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24143591 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22163242 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2298927634944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚120905348754768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 _step‚256 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2233549134848 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22162982 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845031271902792e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚122499767114752 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2233637156864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24143593 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56444739 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚120905331715408 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2226627898592 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32291003 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0½Òqœ ë ÑO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚32291003 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚124733316249600 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8150815 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚124733316249600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14012424 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚124733421310976 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8150558 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2226541750118 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24143591 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚22163242 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2298927634944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚120905348754768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 _step‚256 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2233549134848 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚22162982 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845031271902792e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚122499767114752 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2233637156864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚24143593 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56444739 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚120905331715408 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2226627898592 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145€½0ë×2ì - _wandbruntime‚1281  _runtime‚1281@×CýÜ&í ÖM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32300901 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚122537150521344 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2234227244544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24150912 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32300901 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8153232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚123169524672676 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚120942220822528 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56461972 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24150912 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2234315273728 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22169949 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119352609106944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56461972 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2227217701562 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32311058 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32311058 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2234227244544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14016717 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚124771465795072 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚119352810222592 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 _runtime‚1281.524532381 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2234315273728 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚124771465795072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _step‚257 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚122537133481984 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2227303850148 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24150914 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56451813 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚120942203783168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22169711 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚123169421484730 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751ñ’Ù¢á&î ÛM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56461972 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24150912 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2234315273728 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22169949 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119352609106944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56461972 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2227217701562 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32311058 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32311058 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚235 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2234227244544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14016717 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚124771465795072 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚119352810222592 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 _runtime‚1281.524532381 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2234315273728 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚124771465795072 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _step‚257 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚122537133481984 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2227303850148 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚24150914 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚56451813 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚120942203783168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22169711 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚123169421484730 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32300901 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚122537150521344 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2234227244544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24150912 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32300901 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8153232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚123169524672676 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚120942220822528ÉGr2ï - _wandbruntime‚1281  _runtime‚1281'ž|1ð þ 1loss/FBt2/assignment_loss‚0.16274774082014573 /loss/FRt1/detection_loss‚0.5127380192407268 /loss/FBt1/detection_loss‚1.0115136194788514 /loss/FBt2/detection_loss‚1.5718177686600636 &loss/total_loss‚13.760273773261579 2loss/SRqqt2/assignment_loss‚0.8582677643897542 _step‚258 1loss/FBt1/assignment_loss‚0.16219415109840768 trainer/global_step‚3199 $ _timestamp‚1.784503127462216e+09 1loss/SRqqt1/assignment_loss‚1.018504750368393 .loss/FRt2/detection_loss‚1.370298809472572 1loss/SRqqt1/detection_loss‚1.6918642213260215 1loss/SRqqt2/detection_loss‚0.8393133830999918 epoch‚3 _runtime‚1281.525243092 /loss/FRt1/assignment_loss‚3.105430112466317 0loss/FRt2/assignment_loss‚1.4555834328403343äÙx ñ ƒ /loss/FRt1/assignment_loss‚3.105430112466317 0loss/FRt2/assignment_loss‚1.4555834328403343 1loss/FBt2/assignment_loss‚0.16274774082014573 /loss/FRt1/detection_loss‚0.5127380192407268 /loss/FBt1/detection_loss‚1.0115136194788514 /loss/FBt2/detection_loss‚1.5718177686600636 &loss/total_loss‚13.760273773261579 2loss/SRqqt2/assignment_loss‚0.8582677643897542 _step‚258 1loss/FBt1/assignment_loss‚0.16219415109840768 trainer/global_step‚3199 $ _timestamp‚1.784503127462216e+09 1loss/SRqqt1/assignment_loss‚1.018504750368393 .loss/FRt2/detection_loss‚1.370298809472572 1loss/SRqqt1/detection_loss‚1.6918642213260215 1loss/SRqqt2/detection_loss‚0.8393133830999918 epoch‚3 _runtime‚1281.525243092‚§þá1fò ‚(:Û ߶õÒÈ…ö¿ proc.cpu.threads‚17 network.recv‚401724% disk.nvme0n1.out‚10762.4951171875 disk.dm-1.out‚947.7314453125' proc.memory.availableMB‚ 64.07421875) disk./.usagePercent‚76.12516983695652$ disk./.usageGB‚1168.711669921875! disk.nvme1c1n1.in‚ 10.48046875% disk.nvme1c1n1.out‚1895.736328125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 87.6328125 cpu‚26.488442139504 network.sent‚ 3.988502e+06 disk.nvme0n1.in‚ 87.6328125 disk.nvme1n1.in‚ 10.48046875# disk.nvme1n1.out‚1895.736328125 disk.md0.in‚0 disk.md0.out‚ 35.53515625 disk.dm-0.in‚ 98.11328125# memory_percent‚99.6089220046997! disk.dm-0.out‚11634.443359375 disk.dm-1.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚10762.4951171875# proc.memory.rssMB‚ 2873.37890625* proc.memory.percent‚17.537713050842285Þô4‹šó ‚(: ߶õÒéØÏÀ gpu.0.gpu‚47.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.53' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.17666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Qfé2ô - _wandbruntime‚1298  _runtime‚1298­Ÿ2…œ õ È| 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.27272728085517883 *jet/accuracy_1_of_4‚0.9006295204162598 particle/accuracy_1_of_4‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11175914853811264 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5347589254379272 +jet/accuracy_1_of_1‚0.26264166831970215 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_5‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6360393762588501 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3254479467868805 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.5144737362861633 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.4760684072971344 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9760570526123047 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8430241942405701 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9876543879508972 particle/accuracy_5_of_5‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5357142686843872 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.5892857313156128 $ _timestamp‚1.784503144920104e+09 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9951691627502441 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1752265989780426 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14998942613601685 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9692983031272888 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2142857164144516 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7041115164756775 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13923141360282898 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.982323169708252 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1280391961336136 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9873915910720825 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5504423975944519 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00241485727019608 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.11131798475980759 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6360393762588501 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&PuritrØ5²3y/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10317735373973846 *jet/accuracy_1_of_3‚0.8609361052513123 /particle/accuracy_4_of_4‚0.9201725721359253 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9971088171005249 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14152196049690247 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.2782658338546753 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.031930964440107346 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9022599458694458 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9941833019256592 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.22748753428459167 jet/accuracy_3_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5804782509803772 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16388696432113647 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7878745794296265 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2578522264957428 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09188351035118103 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1418643444776535 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5634215474128723 +jet/accuracy_2_of_3‚0.42449700832366943 jet/accuracy_1_of_5‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6255583167076111 jet/accuracy_4_of_5‚0 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.537534236907959 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09899887442588806 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.11659028381109238 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.16744966804981232 +particle/sensitivity‚0.5753657817840576 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7222222089767456 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9929561614990234 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5533198118209839 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.06354167312383652 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23750440776348114 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23541918396949768 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6460458040237427 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9734490513801575 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.1468288153409958 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_4_of_4‚0.00833333283662796 particle/accuracy_1_of_2‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5357142686843872 *loss/val_total_loss‚13.426396167266148 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.47646626830101013 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6314318776130676 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.27272728085517883 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5786460638046265 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.011971966363489628 .particle/accuracy_3_of_4‚0.981916069984436 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7890350222587585 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 jet/accuracy_2_of_5‚0 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.579028308391571 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5318903923034668 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5117279887199402 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.42873114347457886 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +particle/specificity‚0.9022230505943298 *jet/accuracy_1_of_2‚0.7217035293579102 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3068181574344635 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9201849102973938 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2010592520236969 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5273154973983765 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6329161524772644 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5021576881408691 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 /particle/accuracy_3_of_3‚0.9984782934188843 ( Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9807692170143127 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6410530209541321 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7546253800392151 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.43121033906936646 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5361329913139343 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9907861351966858 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9708035588264465 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.840369701385498 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 (particle/accuracy‚0.8320311307907104 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9167409539222717 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 trainer/global_step‚3203 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5441757440567017 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.520966649055481 particle/accuracy_2_of_3‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4166666865348816 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4930555522441864 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2777777910232544 -"Purity/2FRt0SRqqt1FBt/event_purity‚0.1875 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.11703048646450043 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16821643710136414 +jet/accuracy_3_of_4‚0.05950517952442169 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24546658992767334 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6349824070930481 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.570131778717041 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12500855326652527 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4127604365348816 6validation_average_jet_accuracy‚0.3067636489868164 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.776495635509491 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.11922618001699448 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.967573344707489 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7735588550567627 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.29012346267700195 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9941833019256592 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5528060793876648 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16345925629138947 particle/accuracy_2_of_4‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12916181981563568 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.5952934622764587 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1842973679304123 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.11448907852172852 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2989027500152588 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9952380657196045 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.09225887060165405 _step‚259 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.38811397552490234 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0066850860603153706 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14152196049690247 particle/accuracy_4_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.26549673080444336 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.04166795685887337 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7464590072631836 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3201024830341339 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9647774696350098 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9592809081077576 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1629001349210739 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03707710653543472 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16263626515865326 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9906178712844849 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2777777910232544 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.1875 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5949227809906006 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.535394549369812 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12668539583683014 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.003982129041105509 * Purity/2FRt*SRqqt2FBt/FBt_purity‚0.875 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.49301114678382874 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.790168046951294 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 6 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.998015820980072 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.27272728085517883 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9824514985084534 particle/accuracy_1_of_3‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9351338744163513 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6330238580703735 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4166666865348816 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9285714030265808 *jet/accuracy_2_of_4‚0.6207723617553711 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6114108562469482 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6531368494033813 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14994461834430695 particle/accuracy_2_of_2‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9476496577262878 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5938155055046082 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.12960222363471985 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6548226475715637 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 epoch‚3 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5495060682296753 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5504423975944519 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9828703999519348 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6276518702507019 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.6708333492279053 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9878354668617249 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5591189861297607 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16327325999736786 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.48181337118148804 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.11659028381109238 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.38499999046325684 particle/accuracy_3_of_5‚1 _runtime‚1298.983660822 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9655808806419373 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688R›§)S>ö Í| 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9878354668617249 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.5591189861297607 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.16327325999736786 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.48181337118148804 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.11659028381109238 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.38499999046325684 particle/accuracy_3_of_5‚1 _runtime‚1298.983660822 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9655808806419373 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.27272728085517883 *jet/accuracy_1_of_4‚0.9006295204162598 particle/accuracy_1_of_4‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.7777777910232544 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11175914853811264 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5347589254379272 +jet/accuracy_1_of_1‚0.26264166831970215 particle/accuracy_1_of_1‚1 particle/accuracy_1_of_5‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.6360393762588501 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3254479467868805 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.5144737362861633 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.4760684072971344 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 jet/accuracy_5_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9760570526123047 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8430241942405701 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9876543879508972 particle/accuracy_5_of_5‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5357142686843872 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.5892857313156128 $ _timestamp‚1.784503144920104e+09 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9951691627502441 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1752265989780426 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14998942613601685 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9692983031272888 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.2142857164144516 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7041115164756775 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13923141360282898 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 6 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.982323169708252 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1280391961336136 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9873915910720825 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5504423975944519 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.00241485727019608 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.11131798475980759 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.6360393762588501 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.10317735373973846 *jet/accuracy_1_of_3‚0.8609361052513123 /particle/accuracy_4_of_4‚0.9201725721359253 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9971088171005249 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14152196049690247 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.2782658338546753 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.031930964440107346 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9022599458694458 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9941833019256592 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.22748753428459167 jet/accuracy_3_of_5‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.5804782509803772 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16388696432113647 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.7878745794296265 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.2578522264957428 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09188351035118103 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1418643444776535 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.5634215474128723 +jet/accuracy_2_of_3‚0.42449700832366943 jet/accuracy_1_of_5‚1 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6255583167076111 jet/accuracy_4_of_5‚0 8"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.537534236907959 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09899887442588806 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.11659028381109238 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.16744966804981232 +particle/sensitivity‚0.5753657817840576 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7222222089767456 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9929561614990234 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5533198118209839 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.06354167312383652 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.23750440776348114 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23541918396949768 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.6460458040237427 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9734490513801575 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.1468288153409958 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_4_of_4‚0.00833333283662796 particle/accuracy_1_of_2‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5357142686843872 *loss/val_total_loss‚13.426396167266148 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.47646626830101013 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.6314318776130676 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.27272728085517883 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5786460638046265 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.011971966363489628 .particle/accuracy_3_of_4‚0.981916069984436 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7890350222587585 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 jet/accuracy_2_of_5‚0 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.579028308391571 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.5318903923034668 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5117279887199402 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.42873114347457886 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +particle/specificity‚0.9022230505943298 *jet/accuracy_1_of_2‚0.7217035293579102 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3068181574344635 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9201849102973938 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2010592520236969 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.5273154973983765 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.6329161524772644 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5021576881408691 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 /particle/accuracy_3_of_3‚0.9984782934188843 ( Purity/2FRt*SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9807692170143127 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6410530209541321 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.7546253800392151 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.43121033906936646 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5361329913139343 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9907861351966858 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9708035588264465 8"Purity/0FRt0SRqqt2FBt/event_purity‚0.840369701385498 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 (particle/accuracy‚0.8320311307907104 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9167409539222717 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 trainer/global_step‚3203 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5441757440567017 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.520966649055481 particle/accuracy_2_of_3‚1 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.4166666865348816 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4930555522441864 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2777777910232544 -"Purity/2FRt0SRqqt1FBt/event_purity‚0.1875 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.11703048646450043 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16821643710136414 +jet/accuracy_3_of_4‚0.05950517952442169 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.24546658992767334 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.6349824070930481 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.570131778717041 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12500855326652527 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4127604365348816 6validation_average_jet_accuracy‚0.3067636489868164 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.776495635509491 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.11922618001699448 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.967573344707489 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7735588550567627 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.5714285969734192 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.29012346267700195 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9941833019256592 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5528060793876648 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.16345925629138947 particle/accuracy_2_of_4‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12916181981563568 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 'particle/f_score‚0.5952934622764587 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.1842973679304123 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.11448907852172852 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.2989027500152588 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9952380657196045 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.09225887060165405 _step‚259 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.38811397552490234 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0066850860603153706 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.14152196049690247 particle/accuracy_4_of_5‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.26549673080444336 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.04166795685887337 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7464590072631836 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3201024830341339 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9647774696350098 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9592809081077576 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1629001349210739 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03707710653543472 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16263626515865326 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9906178712844849 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2777777910232544 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.1875 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5949227809906006 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.535394549369812 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12668539583683014 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.003982129041105509 * Purity/2FRt*SRqqt2FBt/FBt_purity‚0.875 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.49301114678382874 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.790168046951294 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 6 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.998015820980072 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.27272728085517883 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9824514985084534 particle/accuracy_1_of_3‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9351338744163513 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.6330238580703735 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.4166666865348816 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9285714030265808 *jet/accuracy_2_of_4‚0.6207723617553711 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6114108562469482 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.6531368494033813 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14994461834430695 particle/accuracy_2_of_2‚1 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9476496577262878 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5938155055046082 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.12960222363471985 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.6548226475715637 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 epoch‚3 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5495060682296753 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5504423975944519 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9828703999519348 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.6276518702507019 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.6708333492279053 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0ƒô_;—p÷ ‚(:å î¶õÒØ£æ¿& disk.nvme0n1.out‚12113.12158203125$ memory_percent‚98.80280494689941 cpu‚26.50203525508392 disk.md0.out‚ 35.90234375 disk.nvme0c0n1.in‚ 87.6328125( disk.nvme0c0n1.out‚12113.12158203125! disk.nvme1c1n1.in‚ 10.48046875% disk.nvme1n1.out‚2320.18408203125 disk.md0.in‚0) proc.memory.percent‚17.53396987915039 network.sent‚ 4.064818e+06' disk.nvme1c1n1.out‚2320.18408203125 disk.nvme1n1.in‚ 10.48046875 disk.dm-0.in‚ 98.11328125 disk.dm-1.in‚0" disk.dm-1.out‚1035.73486328125 disk.rbd0.in‚0 proc.cpu.threads‚17 network.recv‚406417% disk./.usageGB‚1168.6655540466309 disk.nvme0n1.in‚ 87.6328125# disk.dm-0.out‚13320.72705078125 disk.rbd0.out‚0' proc.memory.availableMB‚ 196.1484375! proc.memory.rssMB‚ 2872.765625( disk./.usagePercent‚76.12216603462835m{ ›ø ‚(: î¶õÒ»Š“À gpu.0.gpu‚76.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚116.651' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.88366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êr¿€2ù - _wandbruntime‚1317  _runtime‚1317€6DÈú  trainer/global_step‚3249 $ _timestamp‚1.784503163810137e+09 _runtime‚1317.873278184 _step‚260 ' lr-AdamW/pg1‚0.00011105419923209771 ' lr-AdamW/pg2‚0.00011105419923209771‹Ãž Íû Ç trainer/global_step‚3249 $ _timestamp‚1.784503163810137e+09 _runtime‚1317.873278184 _step‚260 ' lr-AdamW/pg1‚0.00011105419923209771 ' lr-AdamW/pg2‚0.00011105419923209771„vë_2ü - _wandbruntime‚1317  _runtime‚1317‚î=b¸ý ËO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 $ _timestamp‚1.784503163811107e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24747109 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122082098930688 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚127628656265216 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚125340766110720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8353193 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚123717424571760 @ߌ!8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚123717441611120 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127628656265216 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57712275 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚125340749071360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330ÆÃ6,Ö'þ ÐO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚57702130 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚127628656265216 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚57712275 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚125340749071360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 $ _timestamp‚1.784503163811107e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24747109 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚122082098930688 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚127628656265216 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚125340766110720 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8353193 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚123717424571760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚123717441611120 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22680638 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚57719349 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24754432 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2288585303552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57729508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 % _timestamp‚1.7845031640811315e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32975076 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2288585303552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14325297 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚123754296639520 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚119763159725056 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32964919 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2281527333122 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚126035737824056 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚127666805810688 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32975076 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123754313678880 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚125378115438592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚32964919 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122118493967872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚119763184628736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚125378115438592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8355344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2355128932864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 trainer/global_step‚3249 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚125378132477952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚122118288657920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚57729508 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 % _timestamp‚1.7845031640811315e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32975076 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2288585303552 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14325297 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚123754296639520 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚119763159725056 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32964919 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2281527333122 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚126035737824056 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚127666805810688 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32975076 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚123754313678880 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚125378115438592 NADeviceStatsMonitor.on_train_batch_end/allocation.small_p·®qqool.freed‚32964919 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122118493967872 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚119763184628736 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚125378115438592 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8355344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2355128932864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 trainer/global_step‚3249 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚125378132477952 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚122118288657920 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22680638 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0†Õš)J2‚ - _wandbruntime‚1318  _runtime‚1318/&1ƒƒ û &loss/total_loss‚14.480769642689353 epoch‚4 trainer/global_step‚3249 /loss/FBt1/detection_loss‚1.3786541570130955 _runtime‚ 1318.14498629 0loss/FRt2/assignment_loss‚1.4453803039692317 1loss/SRqqt1/assignment_loss‚0.978950769381589 .loss/FRt2/detection_loss‚1.485932501285936 1loss/SRqqt1/detection_loss‚1.3714302257151791 % _timestamp‚1.7845031640821378e+09 _step‚263 /loss/FRt1/assignment_loss‚2.955144691060601 2loss/SRqqt2/assignment_loss‚0.8762716094642716 0loss/FBt1/assignment_loss‚0.1426549861211082 1loss/FBt2/assignment_loss‚0.09746487683232771 /loss/FRt1/detection_loss‚0.5607402073218177 0loss/SRqqt2/detection_loss‚0.745809554163047 .loss/FBt2/detection_loss‚2.4423357603611497}À@„ € 0loss/FRt2/assignment_loss‚1.4453803039692317 1loss/SRqqt1/assignment_loss‚0.978950769381589 .loss/FRt2/detection_loss‚1.485932501285936 1loss/SRqqt1/detection_loss‚1.3714302257151791 % _timestamp‚1.7845031640821378e+09 _step‚263 /loss/FRt1/assignment_loss‚2.955144691060601 2loss/SRqqt2/assignment_loss‚0.8762716094642716 0loss/FBt1/assignment_loss‚0.1426549861211082 1loss/FBt2/assignment_loss‚0.09746487683232771 /loss/FRt1/detection_loss‚0.5607402073218177 0loss/SRqqt2/detection_loss‚0.745809554163047 .loss/FBt2/detection_loss‚2.442335760361149 &loss/total_loss‚14.480769642689353 epoch‚4 trainer/global_step‚3249 /loss/FBt1/detection_loss‚1.3786541570130955 _runtime‚ 1318.14498629‡¶×Ûÿm… ‚(:â ý¶õÒŹó¿% disk.nvme0n1.out‚12129.3623046875! disk.nvme1c1n1.in‚ 10.48046875( proc.memory.availableMB‚ 196.07421875 cpu‚26.495449170323518 proc.cpu.threads‚17 network.sent‚ 4.120898e+06 network.recv‚410438) disk./.usagePercent‚76.11509520967549$ disk./.usageGB‚1168.556999206543 disk.md0.in‚0 disk.dm-0.in‚ 98.11328125 disk.rbd0.out‚0' disk.nvme0c0n1.out‚12129.3623046875 disk.md0.out‚ 36.17578125" disk.dm-1.out‚1075.62158203125 disk.rbd0.in‚0 disk.nvme1n1.in‚ 10.48046875$ disk.nvme1n1.out‚2362.2763671875! proc.memory.rssMB‚ 2872.765625 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 87.6328125 disk.nvme0n1.in‚ 87.6328125& disk.nvme1c1n1.out‚2362.2763671875# memory_percent‚98.8032579421997) proc.memory.percent‚17.53396987915039# disk.dm-0.out‚13338.56689453125÷’\#œ† ‚(:‘ ý¶õÒ²ûžÀ gpu.0.gpu‚63.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚186.114' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚62.038000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò‰Òri‡ ‚(:Þ Œ·õÒ”¥Ç¿" disk.dm-1.out‚1075.83837890625 disk.nvme0c0n1.in‚ 87.6328125$ disk.nvme0n1.out‚12153.650390625' disk.nvme1c1n1.out‚2363.19580078125 disk.nvme1n1.in‚ 10.48046875% disk.nvme1n1.out‚2363.19580078125 disk.md0.out‚ 36.7734375 disk.dm-1.in‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚12153.650390625 disk.dm-0.in‚ 98.11328125 disk.dm-0.out‚13362.30859375( proc.memory.availableMB‚ 196.07421875! proc.memory.rssMB‚ 2872.765625) proc.memory.percent‚17.53396987915039% disk./.usageGB‚1168.5569534301758! disk.nvme1c1n1.in‚ 10.48046875 disk.md0.in‚0 cpu‚26.48828676703831 network.recv‚414845) disk./.usagePercent‚76.11509222798735 disk.rbd0.out‚0 disk.nvme0n1.in‚ 87.6328125# memory_percent‚98.8032579421997 proc.cpu.threads‚17 network.sent‚ 4.170609e+06ûçµÓˆ ‚(:… Œ·õÒȆ…À gpu.0.gpu‚56.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚152.961' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚50.987 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðŽ…›2‰ - _wandbruntime‚1337  _runtime‚1337.ì8üÈŠ  trainer/global_step‚3299 % _timestamp‚1.7845031838956177e+09 _runtime‚ 1337.95876172 _step‚264 ' lr-AdamW/pg1‚0.00011101007503400833 ' lr-AdamW/pg2‚0.00011101007503400833иßÍ‹ Ç ' lr-AdamW/pg1‚0.00011101007503400833 ' lr-AdamW/pg2‚0.00011101007503400833 trainer/global_step‚3299 % _timestamp‚1.7845031838956177e+09 _runtime‚ 1337.95876172 _step‚264ˆl.D2Œ - _wandbruntime‚1337  _runtime‚1337LGQòÒ' ÌO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2321844935680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2389417579520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25113159 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2321932957696 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25113161 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8474989 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚121512247333888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2314680856974 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33451773 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2321932957696 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14535507 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14535504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚123901640009728 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127875622668260 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129530912368640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚123901849521152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚121512222430208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚127209084472320 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330Qþò×'Ž ÑO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14535507 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14535504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚123901640009728 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚127875622668260 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129530912368640 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚123901849521152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚121512222430208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚127209084472320 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2321844935680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2389417579520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25113159 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2321932957696 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25113161 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8474989 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚121512247333888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2314680856974 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33451773 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2321932957696 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152‰¨ —â2 - _wandbruntime‚1338  _runtime‚13386tÅÈ€ ÖM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129568956845568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33461671 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25120482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚121547942491136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚127246450839552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33461671 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚127246433800192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25120480 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚123938242432512 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127913273875410 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚129569061914112 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58592310 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _runtime‚1338.230036014 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25120482 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2322523045376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25120480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125597917066880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125597900027520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚127246450839552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2322523045376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚121547917587456 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2315270659944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58592310 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127913170687464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 trainer/global_step‚3299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚129569061914112 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23017008 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚123938030831104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 % _timestamp‚1.7845031841666043e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58582151 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14539794 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23017272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14539797 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33471828 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2322611074560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2315356808530 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8477214½ÿ'á&‘ ÛM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58592310 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127913170687464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 trainer/global_step‚3299 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚129569061914112 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23017008 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚123938030831104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 % _timestamp‚1.7845031841666043e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58582151 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14539794 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23017272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚14539797 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33471828 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2322611074560 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2315356808530 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8477214 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚129568956845568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33461671 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25120482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚121547942491136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚127246450839552 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33461671 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚127246433800192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25120480 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚123938242432512 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127913273875410 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚129569061914112 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58592310 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _runtime‚1338.230036014 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25120482 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2322523045376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25120480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125597917066880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125597900027520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚127246450839552 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2322523045376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚121547917587456 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2315270659944ŠW%d2’ - _wandbruntime‚1338  _runtime‚1338£f‡¶“ € /loss/FRt1/detection_loss‚0.5778882653001514 &loss/total_loss‚16.327441148362627 0loss/FRt1/assignment_loss‚3.0447834626011465 /loss/FRt2/detection_loss‚1.3749202143417538 0loss/FBt1/assignment_loss‚0.3055273074117124 0loss/FBt2/assignment_loss‚0.1422026205764174 1loss/SRqqt1/detection_loss‚1.5714927205339604 1loss/SRqqt2/detection_loss‚0.7142154994271501 /loss/FBt2/detection_loss‚1.5257630108621854 % _timestamp‚1.7845031841682377e+09 _step‚267 0loss/FRt2/assignment_loss‚1.4510913921254023 2loss/SRqqt1/assignment_loss‚1.0851639836703326 2loss/SRqqt2/assignment_loss‚0.8977674398282542 /loss/FBt1/detection_loss‚3.6366252316841603 epoch‚4 trainer/global_step‚3299 _runtime‚1338.231617805Ã'” ” … trainer/global_step‚3299 _runtime‚1338.231617805 /loss/FRt1/detection_loss‚0.5778882653001514 &loss/total_loss‚16.327441148362627 0loss/FRt1/assignment_loss‚3.0447834626011465 /loss/FRt2/detection_loss‚1.3749202143417538 0loss/FBt1/assignment_loss‚0.3055273074117124 0loss/FBt2/assignment_loss‚0.1422026205764174 1loss/SRqqt1/detection_loss‚1.5714927205339604 1loss/SRqqt2/detection_loss‚0.7142154994271501 /loss/FBt2/detection_loss‚1.5257630108621854 % _timestamp‚1.7845031841682377e+09 _step‚267 0loss/FRt2/assignment_loss‚1.4510913921254023 2loss/SRqqt1/assignment_loss‚1.0851639836703326 2loss/SRqqt2/assignment_loss‚0.8977674398282542 /loss/FBt1/detection_loss‚3.6366252316841603 epoch‚4‹Å+U¹k• ‚(:à ›·õÒ²Á›À disk.nvme1n1.in‚ 10.48046875 disk.md0.in‚0 disk.md0.out‚ 37.29296875$ memory_percent‚98.80344867706299! proc.memory.rssMB‚ 2872.765625 network.sent‚ 4.189361e+06 network.recv‚418079 disk.rbd0.out‚0 disk.nvme0n1.in‚ 87.6328125' disk.nvme1c1n1.out‚2364.14013671875% disk.nvme1n1.out‚2364.14013671875 disk.dm-0.in‚ 98.11328125 disk.dm-1.in‚0 disk.rbd0.in‚0# disk.nvme0n1.out‚12156.33203125! disk.nvme1c1n1.in‚ 10.48046875" disk.dm-1.out‚1075.90087890625 cpu‚26.483806851866035 disk.nvme0c0n1.in‚ 87.6328125" disk.dm-0.out‚13364.7568359375( proc.memory.availableMB‚ 196.04296875) proc.memory.percent‚17.53396987915039 proc.cpu.threads‚17( disk./.usagePercent‚76.1150932218834% disk./.usageGB‚1168.5569686889648% disk.nvme0c0n1.out‚12156.33203125{›ëÛ›– ‚(: ›·õÒ­‰ÄÀ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚183.646' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚61.215333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;sˆ—2— - _wandbruntime‚1357  _runtime‚1357©‡èɘ à ' lr-AdamW/pg2‚0.00011096507075512273 trainer/global_step‚3349 % _timestamp‚1.7845032033236814e+09 _runtime‚1357.386893879 _step‚268 ' lr-AdamW/pg1‚0.00011096507075512273=GõÎΙ È ' lr-AdamW/pg2‚0.00011096507075512273 trainer/global_step‚3349 % _timestamp‚1.7845032033236814e+09 _runtime‚1357.386893879 _step‚268 ' lr-AdamW/pg1‚0.00011096507075512273Œ¹‡Äé2š - _wandbruntime‚1357  _runtime‚1357p‘²8Ò'› ÌO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33958668 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33948525 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25479211 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2424581636608 _step‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25479211 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2355870699520 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129077385794560 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59427734 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23348087 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23347837 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33958668 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129077402833920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14750004 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚127404648387120 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131433168472064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8597833 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25479209 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚125721381515776 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8598080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7845032033246863e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33948525 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2355782677504 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚127404631347760 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2348510332382 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25479209 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚123296980292608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129753055531668 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33958668 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 L?DeviceStatsM•(Œ…$onitor.on_train_batch_start/active.small_pool.freed‚33948525 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25479211 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2424581636608 _step‚269 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25479211 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2355870699520 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129077385794560 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59427734 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23348087 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23347837 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33958668 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129077402833920 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14750004 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚127404648387120 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131433168472064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8597833 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25479209 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚125721381515776 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8598080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7845032033246863e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33948525 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2355782677504 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚127404631347760 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2348510332382 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25479209 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚123296980292608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129753055531668 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129114769201152 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14754294 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2425091316736 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131471318017536 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8600297 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25486532 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23354850 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59455112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2356460787200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129114769201152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131471318017536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚127441503415520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25486530 trainer/global_step‚3349 % _timestamp‚1.7845032038963134e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2356460787200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 195086336 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129790706738818 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59444953 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚123332700353536 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 _step‚270 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129114752161792 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2356548816384 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23354591 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2356548816384 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129114752161792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 219990016 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2349186283938 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59455112 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚127441520454880 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59444953 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33958423 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2349100135352 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33968580 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚04z{á&Ÿ ÛM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25486532 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8600553 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _runtime‚1357.959601919 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25486530 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚33958423 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2356460787200 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129114769201152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131471318017536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚127441503415520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25486530 trainer/global_step‚3349 % _timestamp‚1.7845032038963134e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2356460787200 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 195086336 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129790706738818 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59444953 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚123332700353536 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 _step‚270 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129114752161792 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2356548816384 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23354591 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2356548816384 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129114752161792 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 219990016 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2349186283938 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59455112 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚127441520454880 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59444953 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚33958423 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2349100135352 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33968580 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚129790603550872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129114769201152 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14754294 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2425091316736 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚131471318017536 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8600297 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25486532 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23354850 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚59455112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60290536 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2459550959616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚125081763058688 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172024832 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _step‚273 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2389720419328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133335424575488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125081738155008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14964507 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2459378934784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25845261 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34455420 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23685002 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2389808441344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196928512 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8720498 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚131630488395076 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14964504 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34445277 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8720731 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2382253659316 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2389808441344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _runtime‚1376.844551239 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚130945704156160 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚131630591582910 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133335424575488 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60300681 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23685238 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133335529636864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚130945721195520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚129248234735760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25845261 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2382339807790 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25845259 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845032227815259e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚127541314018304 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚129248251775120 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374î­I×'ª ÑO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2389720419328 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚133335529636864 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2389808441344 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196928512 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8720498 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚131630488395076 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14964504 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚34445277 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8720731 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2382253659316 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2389808441344 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _runtime‚1376.844551239 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚130945704156160 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚131630591582910 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚133335424575488 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚60300681 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23685238 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚133335529636864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚130945721195520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚129248234735760 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25845261 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2382339807790 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25845259 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845032227815259e+09 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚127541314018304 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚129248251775120 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚3399 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚130945721195520 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚130945704156160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25845259 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚127541117089792 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚34445277 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34455420 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚60290536 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2459550959616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚125081763058688 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172024832 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _step‚273 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2389720419328 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚133335424575488 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125081738155008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14964507 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2459378934784 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚25845261 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34455420 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚23685002 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚133373469052416 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23691727 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚133373574120960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚125117458215936 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2382929610760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚130983087562752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2383015759346 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚130983070523392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 _step‚274 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25852580 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚133373574120960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60307755 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2390398529024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34465332 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125117433312256 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2390486558208 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2390398529024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60317914 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚129285106803520 trainer/global_step‚3399 % _timestamp‚1.7845032230386827e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23691972 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60317914 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464b½ ­ ÛM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚130983070523392 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25852582 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚127577711640064 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25852580 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2460075115008 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚130983087562752 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14968794 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34465332 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚131668139602226 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚60307755 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚133373469052416 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23691727 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚133373574120960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚125117458215936 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2382929610760 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚130983087562752 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2383015759346 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚130983070523392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 _step‚274 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25852580 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚133373574120960 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚60307755 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2390398529024 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34465332 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚125117433312256 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2390486558208 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2390398529024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚60317914 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚129285106803520 trainer/global_step‚3399 % _timestamp‚1.7845032230386827e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23691972 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚60317914 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464’0t2® - _wandbruntime‚1377  _runtime‚1377xÒc¯ € 0loss/FRt2/assignment_loss‚1.4222622579091215 1loss/SRqqt1/assignment_loss‚1.013092123484914 2loss/SRqqt2/assignment_loss‚0.9193232418395293 1loss/FBt1/assignment_loss‚0.19996546574736473 1loss/SRqqt1/detection_loss‚1.7973284634518558 trainer/global_step‚3399 2loss/FBt2/assignment_loss‚0.047196939190572405 .loss/FBt1/detection_loss‚1.908222918879361 _runtime‚1377.102596829 _step‚275 &loss/total_loss‚13.671046065471147 0loss/FRt1/assignment_loss‚2.8542772538760572 1loss/SRqqt2/detection_loss‚0.7268889914487312 /loss/FBt2/detection_loss‚0.7594197322982859 $ _timestamp‚1.784503223039614e+09 /loss/FRt1/detection_loss‚0.6297370805588484 /loss/FRt2/detection_loss‚1.3933315967865072 epoch‚4ÅC½ ° … &loss/total_loss‚13.671046065471147 0loss/FRt1/assignment_loss‚2.8542772538760572 1loss/SRqqt2/detection_loss‚0.7268889914487312 /loss/FBt2/detection_loss‚0.7594197322982859 $ _timestamp‚1.784503223039614e+09 /loss/FRt1/detection_loss‚0.6297370805588484 /loss/FRt2/detection_loss‚1.3933315967865072 epoch‚4 0loss/FRt2/assignment_loss‚1.4222622579091215 1loss/SRqqt1/assignment_loss‚1.013092123484914 2loss/SRqqt2/assignment_loss‚0.9193232418395293 1loss/FBt1/assignment_loss‚0.19996546574736473 1loss/SRqqt1/detection_loss‚1.7973284634518558 trainer/global_step‚3399 2loss/FBt2/assignment_loss‚0.047196939190572405 .loss/FBt1/detection_loss‚1.908222918879361 _runtime‚1377.102596829 _step‚275“†»In± ‚(:ã ¹·õÒè˜À disk.rbd0.out‚0 disk.nvme0n1.in‚ 87.6328125& disk.nvme0n1.out‚12168.65185546875' disk.nvme1c1n1.out‚2365.55029296875 disk.md0.out‚ 38.0859375 network.recv‚426026! disk.dm-1.out‚1076.0654296875 disk.nvme1n1.in‚ 10.48046875! proc.memory.rssMB‚ 2872.765625 network.sent‚ 4.3132e+06) disk./.usagePercent‚76.11509396730544 disk.dm-0.in‚ 98.11328125" disk.dm-0.out‚13376.6279296875 disk.nvme0c0n1.in‚ 87.6328125! disk.nvme1c1n1.in‚ 10.48046875% disk.nvme1n1.out‚2365.55029296875 disk.md0.in‚0% disk./.usageGB‚1168.5569801330566( disk.nvme0c0n1.out‚12168.65185546875$ memory_percent‚98.80449771881104( proc.memory.availableMB‚ 195.87109375) proc.memory.percent‚17.53396987915039 cpu‚26.502451088755965 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd0.in‚0yñK囲 ‚(: ¹·õÒÎÔ¾À gpu.0.gpu‚56.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚199.927' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.64233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Dr“ªi³ ‚(:Þ È·õÒÿÏþ¿( disk.nvme0c0n1.out‚12177.28662109375% disk.nvme1c1n1.out‚2366.115234375 disk.nvme1n1.in‚ 10.48046875$ memory_percent‚98.80561828613281 network.sent‚ 4.364911e+06 disk.md0.out‚ 38.41015625 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme1c1n1.in‚ 10.48046875) proc.memory.percent‚17.53396987915039 proc.cpu.threads‚17) disk./.usagePercent‚76.11509620357154$ disk./.usageGB‚1168.557014465332 disk.md0.in‚0# disk.dm-0.out‚13384.97119140625 disk.nvme0n1.in‚ 87.6328125$ proc.memory.availableMB‚195.6875! proc.memory.rssMB‚ 2872.765625 disk.dm-0.in‚ 98.11328125 disk.dm-1.in‚0& disk.nvme0n1.out‚12177.28662109375# disk.nvme1n1.out‚2366.115234375 cpu‚26.510768975561984 network.recv‚430299! disk.dm-1.out‚1076.2138671875 disk.nvme0c0n1.in‚ 87.6328125Ü8¦´ ‚(:… È·õÒ¸æ¿ gpu.0.gpu‚56.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚142.305' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚47.435 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öÜ™12µ - _wandbruntime‚1396  _runtime‚1396²çhæȶ  _step‚276 ' lr-AdamW/pg1‚0.00011087242485238711 ' lr-AdamW/pg2‚0.00011087242485238711 trainer/global_step‚3449 % _timestamp‚1.7845032422335951e+09 _runtime‚ 1396.29673839¬u…Í· Ç ' lr-AdamW/pg1‚0.00011087242485238711 ' lr-AdamW/pg2‚0.00011087242485238711 trainer/global_step‚3449 % _timestamp‚1.7845032422335951e+09 _runtime‚ 1396.29673839 _step‚276”t(ÕO2¸ - _wandbruntime‚1396  _runtime‚1396îeŒÒ'¹ ÌO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15179007 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2423746183168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8844844 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚133508024446318 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 trainer/global_step‚3449 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2423658161152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚135237785740288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24023599 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61153338 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚129360840135680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 % _timestamp‚1.7845032422345045e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚126866496017408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26211309 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34952172 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚131091855163120 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2494539211776 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2416169283198 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 _step‚277 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135237680678912 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219997184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26211309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚135237680678912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15179004 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61163483 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26211311 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2423746183168 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8844844 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚133508024446318 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 trainer/global_step‚3449 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2423658161152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚135237785740288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24023599 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚61153338 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚129360840135680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 % _timestamp‚1.7845032422345045e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚126866496017408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26211309 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34952172 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚131091855163120 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2494539211776 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2416169283198 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 _step‚277 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135237680678912 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219997184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26211309 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚135237680678912 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15179004 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚61163483 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26211311 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/)Z!segment.large_pool.freed‚0•_L2» - _wandbruntime‚1396  _runtime‚1396ŒgvWÚ&¼ ÔM FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24030305 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61170557 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚129397234102784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24030571 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚132851388884992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61170557 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26218632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2416845234754 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚135275830224384 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚131128710191520 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2416759086168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 _step‚278 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26218632 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚132851388884992 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚3449 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61180716 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2495042928128 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2424336270848 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚133545572465634 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2424424300032 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚131128727230880 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2424424300032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚132851405924352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚135275830224384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2495231723008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15183297 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15183294 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129397447801344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚135275725155840 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8847274 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34962084 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 % _timestamp‚1.7845032424926205e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61180716 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34962084 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚126902216078336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚126902191174656 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚132851405924352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2424336270848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816¬í`¼ß&½ ÙM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚3449 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61180716 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2495042928128 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2424336270848 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚133545572465634 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2424424300032 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚131128727230880 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2424424300032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚132851405924352 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚135275830224384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2495231723008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15183297 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15183294 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129397447801344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚135275725155840 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8847274 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34962084 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 % _timestamp‚1.7845032424926205e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚263 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚61180716 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34962084 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚126902216078336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚126902191174656 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚132851405924352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2424336270848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24030305 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61170557 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚129397234102784 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24030571 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚132851388884992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚61170557 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26218632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2416845234754 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚135275830224384 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚131128710191520 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2416759086168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 _step‚278 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26218632 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚132851388884992 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712–œÆçÚ2¾ - _wandbruntime‚1396  _runtime‚1396Ìæ\¿ ÿ 1loss/SRqqt1/detection_loss‚1.4974698431919826 &loss/total_loss‚13.217336195754674 _runtime‚1396.556399502 1loss/SRqqt2/assignment_loss‚0.831205071397379 /loss/FRt2/detection_loss‚1.5637476859479456 1loss/SRqqt2/detection_loss‚0.6730782756397707 /loss/FBt2/detection_loss‚1.2543719475097788 $ _timestamp‚1.784503242493545e+09 0loss/FRt1/assignment_loss‚2.9745252030051272 2loss/SRqqt1/assignment_loss‚0.9858654096664846 0loss/FBt1/assignment_loss‚0.3478942519999388 1loss/FBt2/assignment_loss‚0.11579947788682465 epoch‚4 _step‚279 0loss/FRt2/assignment_loss‚1.2393079724975098 /loss/FRt1/detection_loss‚0.5937317930335311 /loss/FBt1/detection_loss‚1.1403392639784014 trainer/global_step‚3449¯ÊÎj À „ 1loss/SRqqt2/assignment_loss‚0.831205071397379 /loss/FRt2/detection_loss‚1.5637476859479456 1loss/SRqqt2/detection_loss‚0.6730782756397707 /loss/FBt2/detection_loss‚1.2543719475097788 $ _timestamp‚1.784503242493545e+09 0loss/FRt1/assignment_loss‚2.9745252030051272 2loss/SRqqt1/assignment_loss‚0.9858654096664846 0loss/FBt1/assignment_loss‚0.3478942519999388 1loss/FBt2/assignment_loss‚0.11579947788682465 epoch‚4 _step‚279 0loss/FRt2/assignment_loss‚1.2393079724975098 /loss/FRt1/detection_loss‚0.5937317930335311 /loss/FBt1/detection_loss‚1.1403392639784014 trainer/global_step‚3449 1loss/SRqqt1/detection_loss‚1.4974698431919826 &loss/total_loss‚13.217336195754674 _runtime‚1396.556399502—©±äümÁ ‚(:â ×·õÒÌôÕ¿ disk.dm-0.in‚ 98.11328125 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 87.6328125' disk.nvme0c0n1.out‚12199.8798828125( proc.memory.availableMB‚ 195.90234375 network.recv‚433518% disk./.usageGB‚1168.5573806762695! disk.dm-1.out‚1076.2880859375 disk.rbd0.in‚0& disk.nvme1c1n1.out‚2366.8466796875 disk.nvme1n1.in‚ 10.48046875! proc.memory.rssMB‚ 2872.765625) proc.memory.percent‚17.53396987915039 network.sent‚ 4.367875e+06 disk.nvme0n1.in‚ 87.6328125% disk.nvme0n1.out‚12199.8798828125$ memory_percent‚98.80430698394775 proc.cpu.threads‚17" disk.dm-0.out‚13407.5634765625 disk.rbd0.out‚0! disk.nvme1c1n1.in‚ 10.48046875$ disk.nvme1n1.out‚2366.8466796875 cpu‚26.519270112117695) disk./.usagePercent‚76.11512005707667 disk.md0.in‚0 disk.md0.out‚ 38.72265625c+–‚œ ‚(:‘ ×·õÒ攫À gpu.0.gpu‚80.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚198.613' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.20433333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$œkb2à - _wandbruntime‚1415  _runtime‚1415ý—#ÇÄ Á _runtime‚1415.709746959 _step‚280 & lr-AdamW/pg1‚0.0001108247847124493 & lr-AdamW/pg2‚0.0001108247847124493 trainer/global_step‚3499 % _timestamp‚1.7845032616466036e+09¬æÿ!ÌÅ Æ trainer/global_step‚3499 % _timestamp‚1.7845032616466036e+09 _runtime‚1415.709746959 _step‚280 & lr-AdamW/pg1‚0.0001108247847124493 & lr-AdamW/pg2‚0.0001108247847124493˜¸À½2Æ - _wandbruntime‚1415  _runtime‚1415OltðQ$Ç ÌO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 _step‚281 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26577359 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15393504 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚131180763980288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚3499 % _timestamp‚1.7845032616475177e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2457595902976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137139936782336 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚132935441511760 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2449912610132 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚134682357918720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚134682340879360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2457595902976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35448924 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201122816 aMDeviceh˜ StatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚128651253879808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26577361 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚135385457309726 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8968256 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362´Üc®×'È ÑO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2529308977664 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _runtime‚1415.710607809 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62016140 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62026285 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24362001 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚132935458551120 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26577361 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24361760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚134682357918720 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2457683924992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35438781 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚137140041843712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26577359 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚131180562857472 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8968494 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2449998758606 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2457683924992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15393507 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2529485196800 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚134682340879360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚132935441511760 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2449912610132 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚134682357918720 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚134682340879360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2457595902976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚35448924 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201122816 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚128651253879808 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26577361 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚135385457309726 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8968256 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 _step‚281 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26577359 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15393504 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚131180763980288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚3499 % _timestamp‚1.7845032616475177e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2457595902976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚137139936782336 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128686949037056 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35448679 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚134719707246592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2458362041856 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62043518 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 % _timestamp‚1.7845032619047518e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15397794 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26584680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24368707 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26584682 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35448679 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚135422902141096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚132972313579520 trainer/global_step‚3499 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62033359 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2458274012672 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35458836 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚132972330618880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2450674710162 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 _step‚282 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚134719707246592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62033359 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35458836 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2530192867328 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24368464 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26584682 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2458274012672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128686973940736 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62043518 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8970670 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135423005329042 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26584680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15397797 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚2381bÛá&Ë ÛM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 _step‚282 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚134719707246592 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62033359 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35458836 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2530192867328 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24368464 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26584682 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2458274012672 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128686973940736 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62043518 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8970670 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚135423005329042 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26584680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15397797 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128686949037056 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35448679 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚134719707246592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2458362041856 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62043518 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 % _timestamp‚1.7845032619047518e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15397794 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26584680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24368707 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26584682 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35448679 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚135422902141096 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚132972313579520 trainer/global_step‚3499 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚62033359 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2458274012672 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35458836 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚132972330618880 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2450674710162 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946šÁ0æÙ2Ì - _wandbruntime‚1415  _runtime‚1415á2nÆÍ  /loss/FRt2/detection_loss‚1.4964759438769346 &loss/total_loss‚12.840415066630069 trainer/global_step‚3499 _step‚283 2loss/SRqqt2/assignment_loss‚0.7231961703384809 1loss/FBt2/assignment_loss‚0.11763080965439537 /loss/FRt1/assignment_loss‚2.930627727742799 0loss/FRt2/assignment_loss‚1.2048730962112923 2loss/SRqqt1/assignment_loss‚0.8401394175831228 1loss/FBt1/assignment_loss‚0.31528839938509917 /loss/FRt1/detection_loss‚0.5683963451305492 epoch‚4 _runtime‚1415.968581779 1loss/SRqqt1/detection_loss‚1.1970693843111804 1loss/SRqqt2/detection_loss‚0.8557685913677465 /loss/FBt1/detection_loss‚1.6257609951013563 /loss/FBt2/detection_loss‚0.9651881859271125 % _timestamp‚1.7845032619056947e+09Ó: Î † epoch‚4 _runtime‚1415.968581779 1loss/SRqqt1/detection_loss‚1.1970693843111804 1loss/SRqqt2/detection_loss‚0.8557685913677465 /loss/FBt1/detection_loss‚1.6257609951013563 /loss/FBt2/detection_loss‚0.9651881859271125 % _timestamp‚1.7845032619056947e+09 /loss/FRt2/detection_loss‚1.4964759438769346 &loss/total_loss‚12.840415066630069 trainer/global_step‚3499 _step‚283 2loss/SRqqt2/assignment_loss‚0.7231961703384809 1loss/FBt2/assignment_loss‚0.11763080965439537 /loss/FRt1/assignment_loss‚2.930627727742799 0loss/FRt2/assignment_loss‚1.2048730962112923 2loss/SRqqt1/assignment_loss‚0.8401394175831228 1loss/FBt1/assignment_loss‚0.31528839938509917 /loss/FRt1/detection_loss‚0.5683963451305492›pTÌ5Ï ‚(:Þ æ·õÒ×…À$ disk./.usageGB‚1168.55-c47388305664& disk.nvme1c1n1.out‚2367.4052734375 disk.nvme1n1.in‚ 10.48046875$ disk.nvme1n1.out‚2367.4052734375 disk.md0.out‚ 39.0703125 disk.dm-0.in‚ 98.11328125# disk.dm-0.out‚13408.63134765625 disk.md0.in‚0 disk.rbd0.out‚0$ memory_percent‚98.80390167236328% proc.memory.availableMB‚ 195.96875! proc.memory.rssMB‚ 2872.765625 cpu‚26.526872521920183 disk.nvme0c0n1.in‚ 87.6328125& disk.nvme0n1.out‚12201.26513671875) proc.memory.percent‚17.53396987915039 proc.cpu.threads‚17 network.sent‚ 4.444213e+06( disk./.usagePercent‚76.1151205540247( disk.nvme0c0n1.out‚12201.26513671875 disk.nvme0n1.in‚ 87.6328125! disk.nvme1c1n1.in‚ 10.48046875 disk.dm-1.in‚0 disk.dm-1.out‚1076.404296875 disk.rbd0.in‚0 network.recv‚437916¨ìüiœÐ ‚(:‘ æ·õÒÈÝ®À gpu.0.gpu‚62.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚155.913' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚51.971000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á{çé2Ñ - _wandbruntime‚1435  _runtime‚1435ùN]ÉÒ Ã ' lr-AdamW/pg1‚0.00011077626745951586 ' lr-AdamW/pg2‚0.00011077626745951586 trainer/global_step‚3549 % _timestamp‚1.7845032813115654e+09 _runtime‚1435.374777831 _step‚284àÒ†ÎÓ È _runtime‚1435.374777831 _step‚284 ' lr-AdamW/pg1‚0.00011077626745951586 ' lr-AdamW/pg2‚0.00011077626745951586 trainer/global_step‚3549 % _timestamp‚1.7845032813115654e+09œ]âL62Ô - _wandbruntime‚1435  _runtime‚1435±’â=Ò'Õ ÌO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚134779061939120 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚134779044899760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35935533 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139042297947136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26943411 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 _step‚285 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚136550676280320 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26943411 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139042192885760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35935533 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚137262786985300 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚139042192885760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2491533644800 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2483828234014 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2483742085540 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35945676 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚133000449213952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130436036645888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2491621666816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24699719 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15608007 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26943409 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62889087 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 % _timestamp‚1.7845032813126383e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚136550659240960 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26943409 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2564232154624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _runtime‚1435.375783636 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24699469 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9091712 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9091465 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚1372628901731344V¢×'Ö ÑO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚139042297947136 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9091712 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9091465 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚137262890173134 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚134779061939120 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚134779044899760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚35935533 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139042297947136 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26943411 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 _step‚285 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚136550676280320 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26943411 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139042192885760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35935533 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚137262786985300 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚139042192885760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2491533644800 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2483828234014 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2483742085540 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚35945676 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚133000449213952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130436036645888 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2491621666816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚24699719 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15608007 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26943409 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62889087 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 % _timestamp‚1.7845032813126383e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚136550659240960 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26943409 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2564232154624 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _runtime‚1435.375783636 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24699469 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26950730 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26950732 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2564921359360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚137300335004504 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚136588025608192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚133036846151680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚137300438192450 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 trainer/global_step‚3549 _runtime‚1435.972464511 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35945431 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26950730 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9093859 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚134815916967520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24706153 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚133036628258816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989184 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚136588025608192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15612297 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2492299783680 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62896161 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚130471731803136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚139080342431232 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15612294 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2565114348544 % _timestamp‚1.7845032819091122e+09 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚139080342431232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26950732 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35955588 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2484504185570 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62906320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚286 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62906320 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2484418036984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152Âå6™1Ù ÛM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15612297 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2492299783680 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚62896161 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.largõbI°$e_pool.allocated‚130471731803136 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚139080342431232 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15612294 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2565114348544 % _timestamp‚1.7845032819091122e+09 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚139080342431232 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26950732 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35955588 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2484504185570 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62906320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚286 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62906320 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2484418036984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2492211754496 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26950730 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26950732 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2564921359360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚137300335004504 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚136588025608192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚133036846151680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚137300438192450 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 trainer/global_step‚3549 _runtime‚1435.972464511 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35945431 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚26950730 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9093859 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚134815916967520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24706153 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚133036628258816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989184 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚136588025608192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0ž«a™2Ú - _wandbruntime‚1435  _runtime‚1435oåSÎÛ € $ _timestamp‚1.784503281910226e+09 0loss/FBt1/assignment_loss‚0.3212432947950674 1loss/FBt2/assignment_loss‚0.09862819131727149 trainer/global_step‚3549 _runtime‚1435.973337072 _step‚287 0loss/FRt1/assignment_loss‚2.8857858163742285 0loss/FRt2/assignment_loss‚1.2237267969371555 2loss/SRqqt2/assignment_loss‚0.7058497816676259 &loss/total_loss‚12.661555750011978 epoch‚4 2loss/SRqqt1/assignment_loss‚0.8367176992132476 /loss/FRt1/detection_loss‚0.5991414618934556 1loss/SRqqt2/detection_loss‚0.6752510908698637 /loss/FBt1/detection_loss‚1.8223887311623377 /loss/FRt2/detection_loss‚1.4789771335618733 1loss/SRqqt1/detection_loss‚1.2250541134149229 /loss/FBt2/detection_loss‚0.7887916388049282 ÆÀ Ü … /loss/FBt2/detection_loss‚0.7887916388049282 $ _timestamp‚1.784503281910226e+09 0loss/FBt1/assignment_loss‚0.3212432947950674 1loss/FBt2/assignment_loss‚0.09862819131727149 trainer/global_step‚3549 _runtime‚1435.973337072 _step‚287 0loss/FRt1/assignment_loss‚2.8857858163742285 0loss/FRt2/assignment_loss‚1.2237267969371555 2loss/SRqqt2/assignment_loss‚0.7058497816676259 &loss/total_loss‚12.661555750011978 epoch‚4 2loss/SRqqt1/assignment_loss‚0.8367176992132476 /loss/FRt1/detection_loss‚0.5991414618934556 1loss/SRqqt2/detection_loss‚0.6752510908698637 /loss/FBt1/detection_loss‚1.8223887311623377 /loss/FRt2/detection_loss‚1.4789771335618733 1loss/SRqqt1/detection_loss‚1.2250541134149229ŸL€Ù XÝ ‚(:Í õ·õÒŽÀ¿ disk.nvme1n1.in‚ 10.48046875 disk.md0.out‚ 39.3359375) proc.memory.percent‚17.53396987915039 proc.cpu.threads‚17 disk.dm-0.in‚ 98.11328125 disk.dm-0.out‚13419.24609375 disk.rbd0.in‚0! disk.nvme1c1n1.in‚ 10.48046875 disk.nvme1n1.out‚2368.125 disk.md0.in‚0! proc.memory.rssMB‚ 2872.765625 cpu‚26.527647526762976 network.recv‚442096) disk./.usagePercent‚76.11512452960888$ memory_percent‚98.80433082580566 disk.nvme0c0n1.in‚ 87.6328125& disk.nvme0c0n1.out‚12211.822265625 network.sent‚ 4.492772e+06% disk./.usageGB‚1168.5574493408203 disk.dm-1.in‚0 disk.dm-1.out‚1076.478515625 disk.rbd0.out‚0' proc.memory.availableMB‚ 195.8984375 disk.nvme0n1.in‚ 87.6328125$ disk.nvme0n1.out‚12211.822265625 disk.nvme1c1n1.out‚2368.125GÎ ›Þ ‚(: õ·õÒÜâ¿ gpu.0.gpu‚52.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚119.834' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.94466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»ÒÚkhß ‚(:Ý „¸õÒÑŠË¿% disk.nvme1c1n1.out‚2368.900390625 disk.md0.out‚ 39.7890625$ memory_percent‚98.80435466766357) proc.memory.percent‚17.53396987915039 cpu‚26.521065516885468 disk.md0.in‚0% disk./.usageGB‚1168.5575485229492! disk.nvme1c1n1.in‚ 10.48046875 disk.dm-0.in‚ 98.11328125" disk.dm-0.out‚13427.8720703125 disk.dm-1.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17& disk.nvme0c0n1.out‚12220.849609375$ disk.nvme0n1.out‚12220.849609375! disk.dm-1.out‚1076.6728515625( proc.memory.availableMB‚ 195.89453125! proc.memory.rssMB‚ 2872.765625 network.sent‚ 4.555256e+06 network.recv‚446304) disk./.usagePercent‚76.11513098993319 disk.nvme0n1.in‚ 87.6328125 disk.nvme1n1.in‚ 10.48046875# disk.nvme1n1.out‚2368.900390625 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 87.6328125…æç ›à ‚(: „¸õÒ©ŽÀ gpu.0.gpu‚77.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚112.42' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.473333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G»1;2á - _wandbruntime‚1455  _runtime‚1455;ƒTÉâ à ' lr-AdamW/pg1‚0.00011072687387068908 ' lr-AdamW/pg2‚0.00011072687387068908 trainer/global_step‚3599 % _timestamp‚1.7845033014455214e+09 _runtime‚1455.510594713 _step‚288ÛßÎã È _step‚288 ' lr-AdamW/pg1‚0.00011072687387068908 ' lr-AdamW/pg2‚0.00011072687387068908 trainer/global_step‚3599 % _timestamp‚1.7845033014455214e+09 _runtime‚1455.510594713 Û"šä2ä - _wandbruntime‚1455  _runtime‚1455:kµÑ'å ËO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27309461 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2525471386624 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2517571560948 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36432285 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚132220769604608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25037323 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63741744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25037077 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚138418977602560 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36442428 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚136622648287760 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2525559408640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚140944448989184 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚140944554050560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚138418994641920 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63751889 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134819960934912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 trainer/global_step‚3599 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚140944448989184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚132220794508288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 _runtime‚1455.510683013 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚140944554050560 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27309459 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9214816 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚136622665327120 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27309459 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _step‚289 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139140323036542 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36442428 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152¡V•é æ ÐO XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚138418977602560 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 trainer/global_step‚3599 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚140944448989184 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚132220794508288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 _runtime‚1455.510683013 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚140944554050560 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27309459 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9214816 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚136622665327120 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27309459 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _step‚289 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139140323036542 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36442428 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27309461 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2525471386624 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2517571560948 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36432285 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚132220769604608 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25037323 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚63741744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25037077 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚138418977602560 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36442428 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚136622648287760 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2525559408640 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚140944448989184 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚140944554050560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚138418994641920 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚63751889 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚134819960934912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 Iªí0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 $ _timestamp‚1.784503301446257e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚138418994641920 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27309461 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2599371743744 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2517657709422 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9214573 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2525471386624 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15822504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816¡¯U®2ç - _wandbruntime‚1455  _runtime‚1455G,w|Û&è ÕM ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚138456343969792 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25043831 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚136659520355520 trainer/global_step‚3599 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36452340 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15826794 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27316782 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚136659537394880 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚132256464761856 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚63758963 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27316782 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9217286 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚139177871055858 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 $ _timestamp‚1.784503301716502e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36452340 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134856355426304 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2526237525504 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134856562833408 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25044083 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9217037 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36442183 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36442183 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2518333660978 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63758963 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27316780 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140982493466112 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15826797 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚138456361009152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27316780 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2526149496320 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63769122 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2526149496320 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚134856562833408 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25044083 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9217037 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36442183 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚36442183 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2518333660978 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚63758963 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27316780 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚140982493466112 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15826797 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚138456361009152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27316780 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2526149496320 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚63769122 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2526149496320 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25043831 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚136659520355520 trainer/global_step‚3599 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36452340 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15826794 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104¢-¡OÐ2ê - _wandbruntime‚1455  _runtime‚1455µÊ%Bë € &loss/total_loss‚13.949369581271524 0loss/FRt2/assignment_loss‚1.3832516684418632 1loss/SRqqt1/assignment_loss‚1.019030956100989 2loss/FBt2/assignment_loss‚0.029883528441985163 /loss/FRt1/detection_loss‚0.6687123056172448 0loss/SRqqt1/detection_loss‚1.373210389288303 1loss/SRqqt2/detection_loss‚0.8742823287254978 _step‚291 /loss/FRt2/detection_loss‚1.3763642696058696 epoch‚4 trainer/global_step‚3599 % _timestamp‚1.7845033017175333e+09 _runtime‚1455.780545136 0loss/FRt1/assignment_loss‚2.8533328366279602 2loss/SRqqt2/assignment_loss‚0.7062834189253796 1loss/FBt1/assignment_loss‚0.18662988347705323 .loss/FBt1/detection_loss‚2.257707210249573 /loss/FBt2/detection_loss‚1.2206807857698057ÿuj4 ì … .loss/FBt1/detection_loss‚2.257707210249573 /loss/FBt2/detection_loss‚1.2206807857698057 &loss/total_loss‚13.949369581271524 0loss/FRt2/assignment_loss‚1.3832516684418632 1loss/SRqqt1/assignment_loss‚1.019030956100989 2loss/FBt2/assignment_loss‚0.029883528441985163 /loss/FRt1/detection_loss‚0.6687123056172448 0loss/SRqqt1/detection_loss‚1.373210389288303 1loss/SRqqt2/detection_loss‚0.8742823287254978 _step‚291 /loss/FRt2/detection_loss‚1.3763642696058696 epoch‚4 trainer/global_step‚3599 % _timestamp‚1.7845033017175333e+09 _runtime‚1455.780545136 0loss/FRt1/assignment_loss‚2.8533328366279602 2loss/SRqqt2/assignment_loss‚0.7062834189253796 1loss/FBt1/assignment_loss‚0.18662988347705323£[|Chí ‚(:Ý “¸õÒ§½ß¿$ disk.nvme1n1.out‚2369.7197265625 disk.md0.in‚0# memory_percent‚98.8060712814331 disk.dm-1.in‚0! disk.dm-1.out‚1076.7275390625! disk.nvme1c1n1.in‚ 10.48046875( proc.memory.availableMB‚ 195.61328125! proc.memory.rssMB‚ 2872.765625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 87.6328125) proc.memory.percent‚17.53396987915039 cpu‚26.514613725497625 proc.cpu.threads‚17 network.sent‚ 4.561185e+06 network.recv‚449745) disk./.usagePercent‚76.11512477808289 disk.dm-0.in‚ 98.11328125 disk.nvme0n1.in‚ 87.6328125& disk.nvme1c1n1.out‚2369.7197265625 disk.nvme1n1.in‚ 10.48046875 disk.md0.out‚40.0625% disk./.usageGB‚1168.5574531555176" disk.dm-0.out‚13450.3974609375 disk.rbd0.in‚0' disk.nvme0c0n1.out‚12243.2099609375% disk.nvme0n1.out‚12243.2099609375ç·oŸ›î ‚(: “¸õÒÑÆü¿ gpu.0.gpu‚21.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.485' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.16166666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0N[Ù2ï - _wandbruntime‚1475  _runtime‚1475ÿ :Éð à % _timestamp‚1.7845033215295043e+09 _runtime‚1475.592643308 _step‚292 ' lr-AdamW/pg1‚0.00011067660473710755 ' lr-AdamW/pg2‚0.00011067660473710755 trainer/global_step‚3649ØíÎñ È ' lr-AdamW/pg2‚0.00011067660473710755 trainer/global_step‚3649 % _timestamp‚1.7845033215295043e+09 _runtime‚1475.592643308 _step‚292 ' lr-AdamW/pg1‚0.00011067660473710755¤±så?2ò - _wandbruntime‚1475  _runtime‚1475°ýÄwƒó ËO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64604546 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2559497150464 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25375031 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136639686323200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚140287313003520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25374779 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192996352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚142846705092608 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚138466251675760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27675511 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2559409128448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2634351852544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 _step‚293 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚140287295964160 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36939180 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9338024 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚142846810153984 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚134005552370688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2551401036356 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _runtime‚ 1475.59362099 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27675511 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚142846810153984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚140287313003520 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2634158856192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚138466268715120 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16037007 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9337775 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2559497150464 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚142846705092608 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36929037 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36939180 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217900032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27675509 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.sma³û"ÑN ll_pool.freed‚ 2559409128448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27675509 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚141017652712116 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚140287295964160 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64614691 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 trainer/global_step‚3649 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16037004 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2551487184830 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚134005527467008 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚141017755899950 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36929037 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0Ø.§Ö'ô ÐO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚142846705092608 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚138466251675760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27675511 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2559409128448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2634351852544 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 _step‚293 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚140287295964160 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36939180 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9338024 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚142846810153984 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚134005552370688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2551401036356 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 _runtime‚ 1475.59362099 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚27675511 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚142846810153984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚140287313003520 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2634158856192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚138466268715120 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16037007 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9337775 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2559497150464 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚142846705092608 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36929037 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚36939180 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217900032 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27675509 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2559409128448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27675509 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚141017652712116 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚140287295964160 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64614691 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 trainer/global_step‚3649 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16037004 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2551487184830 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚134005527467008 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚141017755899950 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚36929037 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64604546 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2559497150464 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25375031 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚136639686323200 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚140287313003520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25374779 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192996352 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330¥J²%2õ - _wandbruntime‚1475  _runtime‚1475Û ¼}Û&ö ÕM XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚136676082547200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2552163136386 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚294 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2552076987800 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2635040329216 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚138503123743520 _runtime‚1475.865567573 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9340203 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136676287857152 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16041294 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64631924 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 trainer/global_step‚3649 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27682832 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚140324679370752 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2560175267328 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚134041222624256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2634859922944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64621765 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16041297 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64621765 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36938935 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64631924 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36949092 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9340451 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚140324679370752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2560175267328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25381497 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚134041247527936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2560175267328 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚134041222624256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2634859922944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64621765 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16041297 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64621765 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36938935 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64631924 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚36949092 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9340451 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚140324679370752 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2560175267328 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25381497 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚134041247527936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚138503123743520 _runtime‚1475.865567573 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9340203 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚136676287857152 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16041294 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64631924 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 trainer/global_step‚3649 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27682832 M>DeviceStatsMonitor.on_train_­Ómv4batch_end/allocated_bytes.all.peak‚ 6159996928¦ÈFÃ[2ø - _wandbruntime‚1475  _runtime‚1475«×à`ù ÿ /loss/FRt2/assignment_loss‚1.516352102708657 /loss/FRt1/detection_loss‚0.5782503646382536 _step‚295 0loss/FRt1/assignment_loss‚2.8800597161183625 1loss/SRqqt1/assignment_loss‚0.927222356963787 0loss/FBt1/assignment_loss‚0.2171513415469668 1loss/SRqqt1/detection_loss‚1.2765284287262795 1loss/SRqqt2/detection_loss‚0.7628709209651251 /loss/FBt1/detection_loss‚2.3516683507534153 &loss/total_loss‚13.591726711325844 1loss/FBt2/assignment_loss‚0.03026973470976827 /loss/FBt2/detection_loss‚0.8845603254167872 epoch‚4 trainer/global_step‚3649 % _timestamp‚1.7845033218032987e+09 2loss/SRqqt2/assignment_loss‚0.8025551708822104 /loss/FRt2/detection_loss‚1.3642378978962297 _runtime‚1475.866174464£Pv ú „ /loss/FBt1/detection_loss‚2.3516683507534153 &loss/total_loss‚13.591726711325844 1loss/FBt2/assignment_loss‚0.03026973470976827 /loss/FBt2/detection_loss‚0.8845603254167872 epoch‚4 trainer/global_step‚3649 % _timestamp‚1.7845033218032987e+09 2loss/SRqqt2/assignment_loss‚0.8025551708822104 /loss/FRt2/detection_loss‚1.3642378978962297 _runtime‚1475.866174464 /loss/FRt2/assignment_loss‚1.516352102708657 /loss/FRt1/detection_loss‚0.5782503646382536 _step‚295 0loss/FRt1/assignment_loss‚2.8800597161183625 1loss/SRqqt1/assignment_loss‚0.927222356963787 0loss/FBt1/assignment_loss‚0.2171513415469668 1loss/SRqqt1/detection_loss‚1.2765284287262795 1loss/SRqqt2/detection_loss‚0.7628709209651251§9ÐG«lû ‚(:á ¢¸õÒÊ¡–À cpu‚26.50844781877238( disk.nvme0c0n1.out‚12251.66552734375 network.recv‚454096 disk.dm-0.in‚ 98.11328125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme1c1n1.out‚2370.55224609375% disk.nvme1n1.out‚2370.55224609375) disk./.usagePercent‚76.09970597175682! disk.dm-1.out‚1076.8681640625 disk.nvme0n1.in‚ 87.6328125! proc.memory.rssMB‚ 2872.765625) proc.memory.percent‚17.53396987915039 proc.cpu.threads‚17% disk./.usageGB‚1168.3207359313965 disk.md0.out‚40.34375 disk.nvme0c0n1.in‚ 87.6328125$ memory_percent‚98.80499839782715 network.sent‚ 4.635352e+06 disk.md0.in‚0" disk.dm-0.out‚13458.9345703125& disk.nvme0n1.out‚12251.66552734375! disk.nvme1c1n1.in‚ 10.48046875 disk.nvme1n1.in‚ 10.48046875' proc.memory.availableMB‚ 195.7890625ÓyNÈ›ü ‚(: ¢¸õÒ؃ÊÀ gpu.0.gpu‚54.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚136.701' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚45.56699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jþ$É2ý - _wandbruntime‚1495  _runtime‚1495£Ñ+Èþ  ' lr-AdamW/pg1‚0.00011062546086393349 ' lr-AdamW/pg2‚0.00011062546086393349 trainer/global_step‚3699 % _timestamp‚1.7845033415770779e+09 _runtime‚ 1495.64022083 _step‚296÷Ü‚ÄÍÿ Ç _runtime‚ 1495.64022083 _step‚296 ' lr-AdamW/pg1‚0.00011062546086393349 ' lr-AdamW/pg2‚0.00011062546086393349 trainer/global_step‚3699 % _timestamp‚1.7845033415770779e+09¨Åÿ÷2€ - _wandbruntime‚1495  _runtime‚1495DDÖÑ' ËO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37435932 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16251504 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144748961196032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚1495.641146772 _step‚297 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2585230511764 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2593346870272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16251507 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚138459592175104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37435932 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2593434892288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25712106 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65467348 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9460864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135790310233088 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚142155631365120 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚138459391052288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28041561 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28041559 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚142155614325760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37425789 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2585316660238 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28041559 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚142155631365120 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37435932 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16251504 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚144748961196032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚1495.641146772 _step‚297 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2585230511764 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2593346870272 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16251507 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚138459592175104 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚37435932 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2593434892288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25712106 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚65467348 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9460864 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135790310233088 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚142155631365120 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚138459391052288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28041561 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28041559 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚142155614325760 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚37425789 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2585316660238 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330©±ˆ½2ƒ - _wandbruntime‚1495  _runtime‚1495ºä|ßP"„ ÖM FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65494726 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚138495790137344 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135825980486656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚3699 % _timestamp‚1.7845033418457317e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚142932633594728 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚138495991252992 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2585906463208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65484567 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16255794 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚142932736782674 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28048882 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2594024979968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚65484567 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28048880 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚142192997732352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚140346744170880 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2594113009152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 _step‚298 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135826005390336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25718806 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144787110741504 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144787005672960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25719053 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28048880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2669985862656 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2585992611794 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2594113009152 PDeviceStatsMonitor.on_train_batch_end/allo4÷¶Œcated_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚142192980692992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2669809650688 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142192997732352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚140346727131520 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚65494726 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37435687 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0ïÍÄá&… ÛM FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚65494726 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚138495790137344 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135825980486656 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚3699 % _timestamp‚1.7845033418457317e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚142932633594728 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚138495991252992 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2585906463208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚65484567 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16255794 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚142932736782674 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28048882 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2594024979968 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚65484567 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28048880 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚142192997732352 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚140346744170880 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2594113009152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 _step‚298 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚135826005390336 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25718806 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚144787110741504 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚144787005672960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25719053 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28048880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2669985862656 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2585992611794 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2594113009152 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚142192980692992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2669809650688 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142192997732352 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚140346727131520 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚65494726 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37435687 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0ª-6b2† - _wandbruntime‚1495  _runtime‚1495j¯*‡ ý 2loss/SRqqt2/assignment_loss‚0.8447622702013943 /loss/FRt2/detection_loss‚1.3631211593015502 $loss/total_loss‚19.2912569196004 2loss/SRqqt1/assignment_loss‚0.9998277194984914 /loss/FRt1/detection_loss‚0.6356877146772176 1loss/SRqqt1/detection_loss‚1.3554059746150504 1loss/SRqqt2/detection_loss‚0.6969195429448759 .loss/FBt1/detection_loss‚7.943895689050719 /loss/FBt2/detection_loss‚0.8817137022881039 trainer/global_step‚3699 0loss/FBt1/assignment_loss‚0.4714202843955166 1loss/FBt2/assignment_loss‚0.20862765553697174 epoch‚4 % _timestamp‚1.7845033418468165e+09 _runtime‚1495.909687033 /loss/FRt1/assignment_loss‚2.682006279207731 0loss/FRt2/assignment_loss‚1.2078689278827772 _step‚299ù½ˆ ‚ /loss/FRt1/assignment_loss‚2.682006279207731 0loss/FRt2/assignment_loss‚1.2078689278827772 _step‚299 2loss/SRqqt2/assignment_loss‚0.8447622702013943 /loss/FRt2/detection_loss‚1.3631211593015502 $loss/total_loss‚19.2912569196004 2loss/SRqqt1/assignment_loss‚0.9998277194984914 /loss/FRt1/detection_loss‚0.6356877146772176 1loss/SRqqt1/detection_loss‚1.3554059746150504 1loss/SRqqt2/detection_loss‚0.6969195429448759 .loss/FBt1/detection_loss‚7.943895689050719 /loss/FBt2/detection_loss‚0.8817137022881039 trainer/global_step‚3699 0loss/FBt1/assignment_loss‚0.4714202843955166 1loss/FBt2/assignment_loss‚0.20862765553697174 epoch‚4 % _timestamp‚1.7845033418468165e+09 _runtime‚1495.909687033«U¼óîe‰ ‚(:Ú ±¸õÒŽÅÆ¿% disk./.usageGB‚1168.3207511901855$ disk.nvme1c1n1.out‚ 2373.08203125 disk.md0.out‚ 40.57421875 disk.rbd0.in‚0( proc.memory.availableMB‚ 195.74609375 cpu‚26.502277910137565) disk./.usagePercent‚76.09970696565286 disk.nvme0n1.in‚ 87.6328125 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚98.80526065826416 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 87.6328125! disk.nvme1c1n1.in‚ 10.48046875 disk.md0.in‚0! disk.dm-1.out‚1076.8955078125! proc.memory.rssMB‚ 2872.765625) proc.memory.percent‚17.53396987915039 network.sent‚ 4.684921e+06 network.recv‚458219% disk.nvme0c0n1.out‚12265.62890625# disk.nvme0n1.out‚12265.62890625 disk.nvme1n1.in‚ 10.48046875" disk.nvme1n1.out‚ 2373.08203125 disk.dm-0.in‚ 98.11328125" disk.dm-0.out‚13474.8720703125%X휊 ‚(:‘ ±¸õÒûÏ«À gpu.0.gpu‚76.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚199.381' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.46033333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uži‹ ‚(:Þ À¸õÒÿôÑ¿$ disk.nvme0n1.out‚12283.845703125 disk.md0.out‚ 41.00390625! disk.nvme1c1n1.in‚ 10.48046875& disk.nvme1c1n1.out‚2373.7880859375! disk.dm-0.out‚13492.716796875 disk.dm-1.in‚0! disk.dm-1.out‚1077.0419921875! proc.memory.rssMB‚ 2872.765625 network.recv‚462731) disk./.usagePercent‚76.09963316887136% disk./.usageGB‚1168.3196182250977 disk.rbd0.in‚0& disk.nvme0c0n1.out‚12283.845703125$ memory_percent‚98.80678653717041( proc.memory.availableMB‚ 195.49609375 proc.cpu.threads‚17 network.sent‚ 4.748651e+06 disk.nvme0c0n1.in‚ 87.6328125 disk.nvme1n1.in‚ 10.48046875 disk.md0.in‚0 disk.dm-0.in‚ 98.11328125) proc.memory.percent‚17.53396987915039 cpu‚26.49636611383562$ disk.nvme1n1.out‚2373.7880859375 disk.rbd0.out‚0 disk.nvme0n1.in‚ 87.6328125FŽƒ?œŒ ‚(:‘ À¸õÒÃþÑ¿ gpu.0.gpu‚87.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚204.571' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚68.19033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ά‰r2 - _wandbruntime‚1515  _runtime‚1515åí¦†ÈŽ  _runtime‚1515.595535696 _step‚300 ' lr-AdamW/pg1‚0.00011057344307033982 ' lr-AdamW/pg2‚0.00011057344307033982 trainer/global_step‚3749 $ _timestamp‚1.784503361532399e+092 Ð;Í Ç $ _timestamp‚1.784503361532399e+09 _runtime‚1515.595535696 _step‚300 ' lr-AdamW/pg1‚0.00011057344307033982 ' lr-AdamW/pg2‚0.00011057344307033982 trainer/global_step‚3749¬1®z”2 - _wandbruntime‚1515  _runtime‚1515¡<ÈgÐ'‘ ÊO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28407609 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217900032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37922541 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚137575068095488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2627372634112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚144023932687360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚146651322360832 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚146651217299456 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚137575043191808 trainer/global_step‚3749 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚144023949726720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28407609 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217900032 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚37922541 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚137575068095488 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2627372634112 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚144023932687360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚146651322360832 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚146651217299456 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚137575043191808 trainer/global_step‚3749 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚144023949726720 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66357528 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37942596 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2704926056960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚144810169646082 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚144061316093952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66347369 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26056303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37932439 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2628050750976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26056535 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9586009 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚146689366844928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37932439 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚144061299054592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2628050750976 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9586238 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 trainer/global_step‚3749 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚146689366844928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚142190330519520 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2627962721792 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66347369 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚137610738349056 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2704749844992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚142190347558880 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28414932 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚229 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7845033618001993e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚144061316093952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28414932 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37942596 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0Ÿ Êûá&• ÛM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37932439 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2628050750976 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26056535 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9586009 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚146689366844928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37932439 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚144061299054592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2628050750976 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9586238 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 trainer/global_step‚3749 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚146689366844928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚142190330519520 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2627962721792 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66347369 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚137610738349056 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2704749844992 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚142190347558880 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28414932 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚229 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7845033618001993e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚144061316093952 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28414932 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37942596 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66357528 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66357528 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37942596 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2704926056960 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚144810169646082 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚144061316093952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66347369 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26056303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816®Ù@H2– - _wandbruntime‚1515  _runtime‚1515ÁãoŽ— ÿ 0loss/FRt2/assignment_loss‚1.2879285595854935 2loss/SRqqt1/assignment_loss‚0.8608809591984594 2loss/SRqqt2/assignment_loss‚0.7581240589848808 1loss/FBt1/assignment_loss‚0.27220190600189653 /loss/FRt1/detection_loss‚0.5591684995656607 /loss/FBt1/detection_loss‚1.3389294049976503 /loss/FRt1/assignment_loss‚2.805994072139904 1loss/FBt2/assignment_loss‚0.18892286744240994 /loss/FRt2/detection_loss‚1.5901609083751636 epoch‚4 trainer/global_step‚3749 % _timestamp‚1.7845033618012395e+09 1loss/SRqqt1/detection_loss‚1.3056683515307386 1loss/SRqqt2/detection_loss‚0.7780182641395466 .loss/FBt2/detection_loss‚1.125196438347815 &loss/total_loss‚12.871194290309619 _step‚303 _runtime‚ 1515.86406805Cs1 ˜ „ 1loss/FBt2/assignment_loss‚0.18892286744240994 /loss/FRt2/detection_loss‚1.5901609083751636 epoch‚4 trainer/global_step‚3749 % _timestamp‚1.7845033618012395e+09 1loss/SRqqt1/detection_loss‚1.3056683515307386 1loss/SRqqt2/detection_loss‚0.7780182641395466 .loss/FBt2/detection_loss‚1.125196438347815 &loss/total_loss‚12.871194290309619 _step‚303 _runtime‚ 1515.86406805 0loss/FRt2/assignment_loss‚1.2879285595854935 2loss/SRqqt1/assignment_loss‚0.8608809591984594 2loss/SRqqt2/assignment_loss‚0.7581240589848808 1loss/FBt1/assignment_loss‚0.27220190600189653 /loss/FRt1/detection_loss‚0.5591684995656607 /loss/FBt1/detection_loss‚1.3389294049976503 /loss/FRt1/assignment_loss‚2.805994072139904¯ÉÐùi™ ‚(:Þ ϸõÒ£Íà¿) disk./.usagePercent‚76.09963341734539$ disk./.usageGB‚1168.319622039795( disk.nvme0c0n1.out‚12285.92431640625 disk.nvme1n1.in‚ 10.48046875$ memory_percent‚98.80552291870117 proc.cpu.threads‚17 network.recv‚465534 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 87.6328125 disk.nvme0n1.in‚ 87.6328125! disk.nvme1c1n1.in‚ 10.48046875 disk.dm-0.in‚ 98.11328125 disk.dm-1.in‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚12285.92431640625 disk.md0.out‚ 41.36328125 disk.dm-0.out‚ 13494.671875! proc.memory.rssMB‚ 2872.765625 cpu‚26.490244639601126! disk.dm-1.out‚1077.1044921875' disk.nvme1c1n1.out‚2374.49755859375% disk.nvme1n1.out‚2374.49755859375 disk.md0.in‚0& proc.memory.availableMB‚ 195.703125) proc.memory.percent‚17.53396987915039 network.sent‚ 4.75146e+064z«œš ‚(:‘ ϸõÒö²¹À gpu.0.gpu‚75.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚201.777' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.25899999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤×ö©2› - _wandbruntime‚1535  _runtime‚1535œcÖÉœ à ' lr-AdamW/pg1‚0.00011052055218949707 ' lr-AdamW/pg2‚0.00011052055218949707 trainer/global_step‚3799 % _timestamp‚1.7845033811650734e+09 _runtime‚1535.229576994 _step‚304âìÀÎ È ' lr-AdamW/pg2‚0.00011052055218949707 trainer/global_step‚3799 % _timestamp‚1.7845033811650734e+09 _runtime‚1535.229576994 _step‚304 ' lr-AdamW/pg1‚0.00011052055218949707°8N]v2ž - _wandbruntime‚1535  _runtime‚1535äŽö\·Ÿ ÌO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2738945764352 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚143997061839760 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚148553578464256 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26386994 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38419293 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16680504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28773659 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚145892251048960 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2661310375936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139359801054208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145892268088320 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16680507 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28773659 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 _step‚305 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9706487 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38429436 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚142098746818560 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2652889462580 trainer/global_step‚3799 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67192952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2661222353920 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 % _timestamp‚1.7845033811658125e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚145892251048960 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38419293 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚145892268088320 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚142098954232832 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all­X ­.current‚ 207414272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139359825957888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚148553578464256 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67203097 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2652975611054 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148553473402880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2661222353920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28773661ª;)×'  ÑO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38419293 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16680504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28773659 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚145892251048960 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2661310375936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139359801054208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145892268088320 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16680507 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28773659 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 _step‚305 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9706487 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚38429436 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚142098746818560 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2652889462580 trainer/global_step‚3799 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67192952 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2661222353920 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 % _timestamp‚1.7845033811658125e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚145892251048960 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚38419293 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚145892268088320 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚142098954232832 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚139359825957888 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚148553578464256 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67203097 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2652975611054 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148553473402880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2661222353920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28773661 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2738945764352 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚143997061839760 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚148553578464256 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26386994±€a¿2¡ - _wandbruntime‚1535  _runtime‚15358®&ìÜ&¢ ÖM G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67210171 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28780980 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38439348 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28780982 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9708707 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚139395521115136 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28780982 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚148591622948352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2739831075328 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2653565414024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚306 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚146687602509490 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845033814351327e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚142135352190464 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2661900463616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16684797 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16684794 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67210171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28780980 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38439348 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139395496211456 _runtime‚1535.498517343 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9708964 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26393501 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3799 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2739642280448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2661900463616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚146687499321544 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67220330 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38429191 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148591517879808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚145929617416192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚144033950946880 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67220330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚145929634455552×2—Ý£ ÛM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2661900463616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16684797 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16684794 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67210171 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28780980 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38439348 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139395496211456 _runtime‚1535.498517343 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9708964 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26393501 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚3799 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2739642280448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2661900463616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚146687499321544 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67220330 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38429191 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148591517879808 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚145929617416192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚144033950946880 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67220330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚145929634455552 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67210171 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28780980 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38439348 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚28780982 QEDeviceStatsMonitor.on_train_batch_end/inøüÛ¦active_split.small_pool.freed‚9708707 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚139395521115136 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28780982 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚148591622948352 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2739831075328 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2653565414024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚306 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚146687602509490 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845033814351327e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚142135352190464²‡Ê`2¤ - _wandbruntime‚1535  _runtime‚1535uÖ½¥  1loss/FBt1/assignment_loss‚0.18387267380730918 2loss/FBt2/assignment_loss‚0.047822840517267484 0loss/FRt1/assignment_loss‚2.7888320779168843 0loss/SRqqt1/detection_loss‚1.325790333497571 .loss/FBt1/detection_loss‚2.051739752835134 trainer/global_step‚3799 % _timestamp‚1.7845033814362142e+09 /loss/FRt2/detection_loss‚1.3841144493397153 1loss/SRqqt2/detection_loss‚0.7815734134208282 _runtime‚1535.499084684 _step‚307 2loss/SRqqt1/assignment_loss‚0.9001088844932383 2loss/SRqqt2/assignment_loss‚0.7370434326528675 /loss/FRt1/detection_loss‚0.5850310303530631 /loss/FBt2/detection_loss‚0.9636528310485375 &loss/total_loss‚13.036020929097518 epoch‚4 0loss/FRt2/assignment_loss‚1.28643920921510279ØfS ¦ † 0loss/FRt1/assignment_loss‚2.7888320779168843 0loss/SRqqt1/detection_loss‚1.325790333497571 .loss/FBt1/detection_loss‚2.051739752835134 trainer/global_step‚3799 % _timestamp‚1.7845033814362142e+09 /loss/FRt2/detection_loss‚1.3841144493397153 1loss/SRqqt2/detection_loss‚0.7815734134208282 _runtime‚1535.499084684 _step‚307 2loss/SRqqt1/assignment_loss‚0.9001088844932383 2loss/SRqqt2/assignment_loss‚0.7370434326528675 /loss/FRt1/detection_loss‚0.5850310303530631 /loss/FBt2/detection_loss‚0.9636528310485375 &loss/total_loss‚13.036020929097518 epoch‚4 0loss/FRt2/assignment_loss‚1.2864392092151027 1loss/FBt1/assignment_loss‚0.18387267380730918 2loss/FBt2/assignment_loss‚0.047822840517267484³Å·éµn§ ‚(:ã Þ¸õÒñƒö¿' disk.nvme1c1n1.out‚2375.23876953125) proc.memory.percent‚17.53396987915039 proc.cpu.threads‚17 network.sent‚ 4.827781e+06 network.recv‚469736 disk.nvme1n1.in‚ 10.48046875% disk.nvme1n1.out‚2375.23876953125 disk.dm-1.in‚0" disk.dm-1.out‚1077.36767578125! proc.memory.rssMB‚ 2872.765625( disk./.usagePercent‚76.1192750395014# disk.dm-0.out‚13556.29345703125 disk.nvme0c0n1.in‚ 87.6328125 cpu‚26.48455981545497 disk.md0.in‚0 disk.dm-0.in‚ 98.11328125 disk.rbd0.out‚0 disk.nvme0n1.in‚ 87.6328125& disk.nvme0n1.out‚12347.74267578125! disk.nvme1c1n1.in‚ 10.48046875$ memory_percent‚98.80595207214355' proc.memory.availableMB‚ 195.6328125% disk./.usageGB‚1168.6211700439453 disk.md0.out‚ 41.671875 disk.rbd0.in‚0( disk.nvme0c0n1.out‚12347.74267578125|Oö›¨ ‚(: Þ¸õÒÀú¥À gpu.0.gpu‚84.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚123.21' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚41.06999999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EOá¾2© - _wandbruntime‚1555  _runtime‚1555è§ó$Ǫ Á & lr-AdamW/pg1‚0.0001104667890685601 & lr-AdamW/pg2‚0.0001104667890685601 trainer/global_step‚3849 % _timestamp‚1.7845034011687884e+09 _runtime‚1555.231930562 _step‚308‰pk‚Ì« Æ & lr-AdamW/pg1‚0.0001104667890685601 & lr-AdamW/pg2‚0.0001104667890685601 trainer/global_step‚3849 % _timestamp‚1.7845034011687884e+09 _runtime‚1555.231930562 _step‚308´ÙÖJa2¬ - _wandbruntime‚1555  _runtime‚1555©ÍVjÒ'­ ÌO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68065899 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚150455834567680 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2695248117760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2686718937988 trainer/global_step‚3849 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚147760569410560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143918490653696 % _timestamp‚1.7845034011697536e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150455729506304 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26724578 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚145840682267120 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38916045 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29139709 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68055754 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚147760586449920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38926188 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚147760586449920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 _step‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚147760569410560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141144583820288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9829327 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1®·×'® ÑO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚150455729506304 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26724578 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚145840682267120 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38916045 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29139709 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚68055754 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚147760586449920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚38926188 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚147760586449920 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 _step‚309 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚147760569410560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141144583820288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9829327 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚68065899 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚150455834567680 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2695248117760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2686718937988 trainer/global_step‚3849 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚147760569410560 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143918490653696 % _timestamp‚1.7845034011697536e+09 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38925943 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚1555.503680317 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38936100 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚150493879051776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141180278977536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38925943 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26731306 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68083132 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38936100 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143955091151360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141180254073856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9831751 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚148565035372898 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚147797952817152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2774812173824 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 trainer/global_step‚3849 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9832009 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚150493773983232 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150493879051776 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143954881647104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2687394889432 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68072973 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29147032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68083132 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚147797935777792 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2774627573248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29147032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2695926234624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2695838205440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16899294 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 # _timestamp‚1.78450340144032e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26731045 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16899297 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚147797952817152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29147030 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _step‚310 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2695838205440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0f·øÿß&± ÙM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚148564932184952 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚145877554334880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2695838205440 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16899294 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 # _timestamp‚1.78450340144032e+09 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26731045 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16899297 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚147797952817152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29147030 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _step‚310 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2695838205440 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29147030 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2687481038018 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚147797935777792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚38925943 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚1555.503680317 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38936100 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚150493879051776 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚141180278977536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚38925943 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚26731306 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68083132 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38936100 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143955091151360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141180254073856 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9831751 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚148565035372898 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚147797952817152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2774812173824 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 trainer/global_step‚3849 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9832009 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚150493773983232 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚150493879051776 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143954881647104 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2687394889432 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68072973 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29147032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68083132 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚147797935777792 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2774627573248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29147032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2695926234624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152357985609728 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39412797 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2809066146816 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚147684285655120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17109504 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39422940 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2729185859584 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2729185859584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2808887830528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚150404817029156 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _step‚313 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39412797 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2809066146816 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚147684285655120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17109504 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39422940 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2729185859584 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2729185859584 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2808887830528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚150404817029156 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _step‚313 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152357985609728 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27068526 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚145774806209024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚149666254139392 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29513082 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚234 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17113794 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68935775 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2729863976448 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68935775 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚152396135155200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68945934 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 _step‚314 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29513080 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29513082 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚147721157722880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚149666271178752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17113797 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚149666254139392 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39432852 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚150442468236306 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚147721140683520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152396135155200 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142965036839936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚150442365048360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚3899 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39422695 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39422695 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68945934 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2809586865664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29513080 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9954963 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚149666271178752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2729863976448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _runtime‚1575.216435582 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚145774598801920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2729775947264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 % _timestamp‚1.7845034211531086e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2721224364840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2gZQ@á&Á ÛM L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152396030086656 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27068526 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚145774806209024 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚149666254139392 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29513082 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚234 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17113794 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚68935775 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2729863976448 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚68935775 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚152396135155200 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚68945934 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 _step‚314 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29513080 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29513082 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚147721157722880 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚149666271178752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17113797 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚149666254139392 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚39432852 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚150442468236306 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚147721140683520 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152396135155200 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142965036839936 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚150442365048360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚3899 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39422695 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39422695 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚68945934 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2809586865664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29513080 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9954963 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚149666271178752 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2729863976448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _runtime‚1575.216435582 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚145774598801920 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2729775947264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 % _timestamp‚1.7845034211531086e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2721224364840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚154260241713152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27400859 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚154260346774528 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2763035579392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2754377888804 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10076852 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚144714074641408 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39909549 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚154260241713152 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚152282353080398 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149527889043120 % _timestamp‚1.7845034406302054e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2763123601408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17324004 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚154260346774528 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚151497206133760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚144714099545088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151497223173120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17324007 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29871809 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151497206133760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149527872003760 trainer/global_step‚3949 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151497223173120 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2843995523072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2763123601408 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69791503 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2843817206784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39909549 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 _runtime‚1594.693348119 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29871811 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29871811 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482@–:Ù'Ì ÓO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚151497223173120 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2843995523072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2763123601408 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69791503 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2843817206784 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39909549 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 _runtime‚1594.693348119 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29871811 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29871811 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚147558095068160 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚147557891848192 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39919692 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚154260241713152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27400859 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚154260346774528 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2763035579392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2754377888804 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10076852 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚144714074641408 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39909549 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚154260241713152 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚152282353080398 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149527889043120 % _timestamp‚1.7845034406302054e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2763123601408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17324004 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚154260346774528 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚151497206133760 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚144714099545088 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚151497223173120 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17324007 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29871809 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚151497206133760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149527872003760 trainer/global_step‚3949½Wu®‚2Í - _wandbruntime‚1594  _runtime‚1594‘+·{Þ&Î ØM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151534589540352 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69798577 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7845034408993716e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2755053840248 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29879130 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚154298391258624 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2763801718272 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39919447 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚147594283279872 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151534572500992 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2763713689088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2763713689088 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39929604 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚152319797911768 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 trainer/global_step‚3949 _step‚318 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69808736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69798577 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚151534572500992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2755139988834 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27407340 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚151534589540352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10079296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2844513481216 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚152319901099714 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17328294ëÔÏ ÝM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚154298391258624 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2763801718272 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39919447 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚147594283279872 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚151534572500992 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2763713689088 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2763713689088 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39929604 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚152319797911768 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 trainer/global_step‚3949 _step‚318 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69808736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚69798577 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚151534572500992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2755139988834 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27407340 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚151534589540352 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10079296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2844513481216 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚152319901099714 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17328294 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚151534589540352 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚69798577 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7845034408993716e+09 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2755053840248 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29879130¾¨]]d2Ð - _wandbruntime‚1594  _runtime‚1594Æÿ5ÕÑ € 2loss/SRqqt2/assignment_loss‚0.8540856799861584 /loss/FRt1/detection_loss‚0.5718060213844631 _step‚319 /loss/FRt2/detection_loss‚1.5161666943340228 1loss/SRqqt2/detection_loss‚0.7423120025627765 epoch‚4 _runtime‚1594.963158927 0loss/FRt2/assignment_loss‚1.5102258767721788 2loss/SRqqt1/assignment_loss‚0.8478912210468059 .loss/FBt1/detection_loss‚1.140910511556567 /loss/FBt2/detection_loss‚0.8104578060273899 % _timestamp‚1.7845034409003255e+09 0loss/FRt1/assignment_loss‚2.7677643750614362 1loss/FBt1/assignment_loss‚0.36388329310418993 0loss/FBt2/assignment_loss‚0.1020108805877873 1loss/SRqqt1/detection_loss‚1.2901581232468564 &loss/total_loss‚12.517672485670634 trainer/global_step‚3949ú  Ò … /loss/FBt2/detection_loss‚0.8104578060273899 % _timestamp‚1.7845034409003255e+09 0loss/FRt1/assignment_loss‚2.7677643750614362 1loss/FBt1/assignment_loss‚0.36388329310418993 0loss/FBt2/assignment_loss‚0.1020108805877873 1loss/SRqqt1/detection_loss‚1.2901581232468564 &loss/total_loss‚12.517672485670634 trainer/global_step‚3949 2loss/SRqqt2/assignment_loss‚0.8540856799861584 /loss/FRt1/detection_loss‚0.5718060213844631 _step‚319 /loss/FRt2/detection_loss‚1.5161666943340228 1loss/SRqqt2/detection_loss‚0.7423120025627765 epoch‚4 _runtime‚1594.963158927 0loss/FRt2/assignment_loss‚1.5102258767721788 2loss/SRqqt1/assignment_loss‚0.8478912210468059 .loss/FBt1/detection_loss‚1.140910511556567¿ÁMˆaÓ ‚(:Ö š¹õÒ“ëî¿ network.recv‚486527 disk.nvme0n1.in‚ 87.6328125" disk.nvme0n1.out‚ 12539.8828125! disk.nvme1c1n1.in‚ 10.48046875$ disk.nvme1n1.out‚2441.7490234375$ memory_percent‚99.20163154602051' proc.memory.availableMB‚ 130.8046875! proc.memory.rssMB‚ 2872.765625 disk.dm-1.in‚0 disk.dm-1.out‚ 1077.7421875 disk.rbd0.in‚0& disk.nvme1c1n1.out‚2441.7490234375 cpu‚26.4690752633172) disk./.usagePercent‚76.08410105839593% disk./.usageGB‚1168.0811614990234 disk.dm-0.in‚ 98.11328125" disk.dm-0.out‚13811.7587890625 disk.nvme1n1.in‚ 10.48046875 disk.md0.out‚ 42.96484375) proc.memory.percent‚17.53396987915039 proc.cpu.threads‚17 network.sent‚ 5.042694e+06 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 87.6328125$ disk.nvme0c0n1.out‚ 12539.8828125 disk.md0.in‚0#µ]ƒÔ ‚(:„ š¹õÒ´ì À gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚200.079' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚66.693 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0STãm2Õ - _wandbruntime‚1614  _runtime‚1614˜*ZÉÖ Ã % _timestamp‚1.7845034605797296e+09 _runtime‚1614.642894618 _step‚320 ' lr-AdamW/pg1‚0.00011030027494620924 ' lr-AdamW/pg2‚0.00011030027494620924 trainer/global_step‚3999ŽŽŸrÎ× È % _timestamp‚1.7845034605797296e+09 _runtime‚1614.642894618 _step‚320 ' lr-AdamW/pg1‚0.00011030027494620924 ' lr-AdamW/pg2‚0.00011030027494620924 trainer/global_step‚3999ÀÑ ¯2Ø - _wandbruntime‚1614  _runtime‚1614ŸÎ^Ó'Ù ÍO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40406301 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2797061343232 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40416444 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2796973321216 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27738971 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30237859 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40406301 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2878963301888 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17538507 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 _runtime‚1614.647618817 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40416444 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚154159682755972 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156162497816576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚151371475391760 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚153365541534720 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚153365524495360 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30237861 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70654305 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 $ _timestamp‚1.784503460580744e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10200464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚146498832503808 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2796973321216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚151371492431120 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2788207364212 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚154159785943806 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10200195 _step‚321 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚153365524495360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149377607003648 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156162497816576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2788293512686 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚156162602877952 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚146498857407488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17538507 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 _runtime‚1614.647618817 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40416444 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚154159682755972 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚156162497816576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚151371475391760 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚153365541534720 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚153365524495360 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚30237861 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚70654305 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 $ _timestamp‚1.784503460580744e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10200464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚146498832503808 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2796973321216 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚151371492431120 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2788207364212 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚154159785943806 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10200195 _step‚321 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚153365524495360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚149377607003648 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156162497816576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2788293512686 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374Á¥×6Y2Û - _wandbruntime‚1614  _runtime‚1614Œt¨ÎÅÜ ×M NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30245182 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40426356 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27745793 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚151408347459520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156200647362048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚153402907901952 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70671538 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27745528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2788969464242 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.784503460850421e+09 _runtime‚1614.913723428 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30245180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 trainer/global_step‚3999 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30245182 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚322 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚156200542293504 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚156200647362048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2879474100224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚153402890862592”›+–â&Ý ÜM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149414204974080 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149414001761280 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚70671538 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27745528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2788969464242 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 $ _timestamp‚1.784503460850421e+09 _runtime‚1614.913723428 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30245180 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 trainer/global_step‚3999 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚30245182 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚322 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚156200542293504 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚156200647362048 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2879474100224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚153402890862592 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30245182 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40426356 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27745793 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚151408347459520 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156200647362048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚153402907901952 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9802631735801697 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08152617514133453 % _timestamp‚1.7845034799427617e+09 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14315903186798096 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 'particle/f_score‚0.6192758679389954 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14114396274089813 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1112140342593193 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7445261478424072 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9943798780441284 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9807692170143127 *jet/accuracy_1_of_3‚0.9116653203964233 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1212267056107521 +particle/specificity‚0.9057459831237793 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7635530829429626 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6510916948318481 +jet/accuracy_1_of_1‚0.28507962822914124 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.22318074107170105 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9930777549743652 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7142786979675293 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7221304774284363 particle/accuracy_5_of_5‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006833448074758053 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7599999904632568 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3355139195919037 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9936763048171997 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5300784707069397 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9482496976852417 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9758642911911011 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7721543908119202 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.42734432220458984 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5717568397521973 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.570197582244873 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4322916567325592 jet/accuracy_5_of_5‚0 _step‚324 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6459701061248779 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.38708776235580444 +jet/accuracy_3_of_3‚0.05461824685335159 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8112834095954895 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1518910825252533 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.354938268661499 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9776326417922974 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900690913200378 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.509829044342041 trainer/global_step‚4004 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9964348673820496 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_2_of_2‚0.9998950958251953 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.590093731880188 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12565124034881592 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5962942838668823 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.992369532585144 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9949070811271667 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6237903833389282 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14178654551506042 particle/accuracy_1_of_4‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7658119201660156 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1228412315249443 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13111597299575806 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7172079682350159 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1377963274717331 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 (particle/accuracy‚0.8407700061798096 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3928571343421936 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.039532359689474106 +jet/accuracy_3_of_4‚0.07300523668527603 -"Purity/2FRt0SRqqt1FBt/event_purity‚0.1875 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7526572346687317 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.7916666269302368 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9825779795646667 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9603335857391357 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3546273708343506 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.07916666567325592 ,jet/accuracy_4_of_4‚0.006349206902086735 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3928571343421936 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6893389821052551 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 *jet/accuracy_1_of_4‚0.9534193873405457 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 particle/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6525290012359619 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.002332239644601941 particle/accuracy_2_of_4‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.35589510202407837 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_3‚1 .particle/accuracy_3_of_3‚0.999514102935791 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7533267140388489 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5942177772521973 7"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.60081946849823 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 +jet/accuracy_2_of_2‚0.12358365952968597 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6023572683334351 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9936224818229675 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9930777549743652 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5751835703849792 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +particle/sensitivity‚0.6031597852706909 6validation_average_jet_accuracy‚0.3310045599937439 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3462910056114197 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1086140051484108 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6867241263389587 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12274853140115738 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004442161414772272 particle/accuracy_1_of_1‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7833333015441895 jet/accuracy_4_of_5‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5994130969047546 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6050975918769836 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.19939656555652618 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1292281597852707 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7222222089767456 epoch‚4 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1336803138256073 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9924675226211548 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24230487644672394 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9973545074462891 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_5‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5429655909538269 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7733483910560608 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.41499999165534973 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9791511297225952 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9898989200592041 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9481579065322876 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9838350415229797 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.47267118096351624 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.34458523988723755 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.540892481803894 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7444466948509216 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5602580904960632 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.41839393973350525 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7599999904632568 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.5460526943206787 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5780519843101501 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6050975918769836 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6562786102294922 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9897854924201965 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0¢Bwá.0127883180975914 particle/accuracy_1_of_2‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 *loss/val_total_loss‚12.297790426888744 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_2_of_3‚0.47726282477378845 *jet/accuracy_2_of_4‚0.7754044532775879 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4618055522441864 particle/accuracy_4_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12135802954435349 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9985119104385376J(«å6>å °| & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8112834095954895 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1518910825252533 8"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.354938268661499 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9776326417922974 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9900690913200378 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.509829044342041 trainer/global_step‚4004 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9964348673820496 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_2_of_2‚0.9998950958251953 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.590093731880188 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_5‚1 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12565124034881592 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5962942838668823 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.992369532585144 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9949070811271667 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6237903833389282 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.14178654551506042 particle/accuracy_1_of_4‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7658119201660156 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.1228412315249443 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13111597299575806 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7172079682350159 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1377963274717331 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 (particle/accuracy‚0.8407700061798096 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3928571343421936 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.039532359689474106 +jet/accuracy_3_of_4‚0.07300523668527603 -"Purity/2FRt0SRqqt1FBt/event_purity‚0.1875 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7526572346687317 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.7916666269302368 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.2222222238779068 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9825779795646667 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9603335857391357 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3546273708343506 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.07916666567325592 ,jet/accuracy_4_of_4‚0.006349206902086735 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3928571343421936 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6893389821052551 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 *jet/accuracy_1_of_4‚0.9534193873405457 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 particle/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6525290012359619 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.002332239644601941 particle/accuracy_2_of_4‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.35589510202407837 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_3‚1 .particle/accuracy_3_of_3‚0.999514102935791 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.7533267140388489 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5942177772521973 7"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.60081946849823 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 +jet/accuracy_2_of_2‚0.12358365952968597 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6023572683334351 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9936224818229675 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9930777549743652 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.5751835703849792 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +particle/sensitivity‚0.6031597852706909 6validation_average_jet_accuracy‚0.3310045599937439 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3462910056114197 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1086140051484108 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6867241263389587 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12274853140115738 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004442161414772272 particle/accuracy_1_of_1‚1 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7833333015441895 jet/accuracy_4_of_5‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5994130969047546 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6050975918769836 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.19939656555652618 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1292281597852707 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7222222089767456 epoch‚4 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1336803138256073 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9924675226211548 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24230487644672394 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9973545074462891 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_5‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5429655909538269 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7733483910560608 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.41499999165534973 -"Purity/2FRt*SRqqt2FBt/event_purity‚0.3125 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9791511297225952 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9898989200592041 particle/accuracy_3_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9481579065322876 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9838350415229797 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.47267118096351624 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.34458523988723755 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.540892481803894 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7444466948509216 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5602580904960632 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.41839393973350525 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.2222222238779068 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7599999904632568 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.5460526943206787 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5780519843101501 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6050975918769836 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6562786102294922 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9897854924201965 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_2‚1 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 *loss/val_total_loss‚12.297790426888744 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 +jet/accuracy_2_of_3‚0.47726282477378845 *jet/accuracy_2_of_4‚0.7754044532775879 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.4618055522441864 particle/accuracy_4_of_5‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12135802954435349 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9985119104385376 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8112834095954895 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6508172154426575 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13319966197013855 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09734123200178146 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7963659167289734 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.8417873382568359 7"Purity/0FRt0SRqqt2FBt/event_purity‚0.89649897813797 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1112763062119484 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13111597299575806 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6071548461914062 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2595248222351074 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8228343725204468 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7097877860069275 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5006788969039917 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.08210211992263794 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.6913132667541504 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9743361473083496 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.012951376847922802 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1870199590921402 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.32499998807907104 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9754272699356079 jet/accuracy_2_of_5‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.07539180666208267 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.46750953793525696 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.15460513532161713 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9786720275878906 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.054446250200271606 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.7678571343421936 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6548418998718262 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.37878790497779846 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10209333151578903 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.030447300523519516 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.10542921721935272 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4231705963611603 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.12388088554143906 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8628357648849487 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.07509928196668625 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.054446250200271606 /particle/accuracy_4_of_4‚0.9475338459014893 _runtime‚1634.006567738 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8080617785453796 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9353630542755127 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 6 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.172461599111557 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.32499998807907104 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8611111044883728 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9802631735801697 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.3571428656578064 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.08152617514133453 % _timestamp‚1.7845034799427617e+09 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.14315903186798096 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 'particle/f_score‚0.6192758679389954 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.14114396274089813 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1112140342593193 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7445261478424072 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9943798780441284 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9807692170143127 *jet/accuracy_1_of_3‚0.9116653203964233 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1212267056107521 +particle/specificity‚0.9057459831237793 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7635530829429626 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6510916948318481 +jet/accuracy_1_of_1‚0.28507962822914124 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.22318074107170105 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9930777549743652 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7142786979675293 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7221304774284363 particle/accuracy_5_of_5‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006833448074758053 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7599999904632568 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.3355139195919037 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9936763048171997 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5300784707069397 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9482496976852417 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9758642911911011 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7721543908119202 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.42734432220458984 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5717568397521973 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.570197582244873 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4322916567325592 jet/accuracy_5_of_5‚0 _step‚324 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6459701061248779 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.38708776235580444 +jet/accuracy_3_of_3‚0.05461824685335159Ä$pâ1næ ‚(:ã ¸¹õÒÏÌ¿ disk.nvme0c0n1.in‚ 87.6328125 disk.md0.out‚ 43.66015625 disk.dm-0.in‚ 98.11328125& proc.memory.availableMB‚ 195.234375 disk.rbd0.out‚0 disk.nvme0n1.in‚ 87.6328125& disk.nvme1c1n1.out‚2508.1220703125$ disk.nvme1n1.out‚2508.1220703125 proc.cpu.threads‚17 network.recv‚494913% disk./.usageGB‚1168.0809707641602! disk.nvme1c1n1.in‚ 10.48046875$ memory_percent‚98.80838394165039" proc.memory.rssMB‚ 2873.2890625 network.sent‚ 5.167673e+06( disk.nvme0c0n1.out‚12548.39794921875& disk.nvme0n1.out‚12548.39794921875 disk.nvme1n1.in‚ 10.48046875 disk.md0.in‚0# disk.dm-0.out‚13821.09423828125) proc.memory.percent‚17.53716468811035 cpu‚26.524526727413026) disk./.usagePercent‚76.08408863469533 disk.dm-1.in‚0 disk.dm-1.out‚1141.779296875 disk.rbd0.in‚0´b ›ç ‚(: ¸¹õÒ«§À gpu.0.gpu‚27.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚200.785' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.92833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ¥¤Xjè ‚(:ß ǹõÒ¥«ó¿% disk./.usageGB‚1167.8437385559082 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 87.6328125& disk.nvme0n1.out‚12555.62548828125! disk.nvme1c1n1.in‚ 10.48046875 disk.dm-0.in‚ 98.11328125 disk.dm-1.in‚0 network.recv‚498755( disk.nvme0c0n1.out‚12555.62548828125 disk.nvme1n1.in‚ 10.48046875 disk.md0.in‚0' proc.memory.availableMB‚ 195.4453125 cpu‚26.530755449868522 network.sent‚ 5.226689e+06) disk./.usagePercent‚76.06863628437766! disk.nvme1n1.out‚ 2548.8828125# disk.dm-0.out‚13827.83837890625$ memory_percent‚98.80709648132324# proc.memory.rssMB‚ 2873.30078125 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.nvme0n1.in‚ 87.6328125# disk.nvme1c1n1.out‚ 2548.8828125 disk.md0.out‚ 44.1796875 disk.dm-1.out‚1181.908203125* proc.memory.percent‚17.537236213684082“ÁÐîœé ‚(:‘ ǹõÒü›À gpu.0.gpu‚89.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚192.794' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚64.26466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/c ¿2ê - _wandbruntime‚1652  _runtime‚1652ë—MÉë à ' lr-AdamW/pg1‚0.00011024303161564906 ' lr-AdamW/pg2‚0.00011024303161564906 trainer/global_step‚4049 % _timestamp‚1.7845034986588433e+09 _runtime‚1652.721969103 _step‚325ƒ¶x Îì È % _timestamp‚1.7845034986588433e+09 _runtime‚1652.721969103 _step‚325 ' lr-AdamW/pg1‚0.00011024303161564906 ' lr-AdamW/pg2‚0.00011024303161564906 trainer/global_step‚4049ÅÔ¢`¥2í - _wandbruntime‚1652  _runtime‚1652@ÚÑÔ'î ÎO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28250984 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30841379 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41080462 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71921841 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41070319 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2851419402240 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2934218435072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159057837832192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10403664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28250748 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30841377 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71911696 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156206523491328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 _step‚326 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152143232065024 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159057837832192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚149209013629952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2934394654208 % _timestamp‚1.7845034986598341e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2851331380224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚156206506451968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚157026102283132 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156206506451968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17847084 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0;\å’· ï ÓO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚154183585287472 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41070319 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201122816 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚154183568248112 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2842516995660 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚159057942893568 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156206523491328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚157025999095298 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2842430847186 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚159057942893568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G>DeviceStatsMonitor1Ò±¢".on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156206506451968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17847084 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28250984 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚30841379 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41080462 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71921841 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41070319 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2851419402240 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2934218435072 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚159057837832192 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10403664 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28250748 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30841377 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71911696 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156206523491328 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 _step‚326 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152143232065024 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159057837832192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚149209013629952 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2934394654208 % _timestamp‚1.7845034986598341e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2851331380224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚156206506451968 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚157026102283132 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159095882309120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚154220457355232 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71939074 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2843106798630 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚327 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚159095987377664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71939074 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28257540 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30848698 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2935101528064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156243889858560 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17851377 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2843192947216 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 % _timestamp‚1.7845034989264762e+09 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156243889858560 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚156243872819200 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152179835218944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30848698 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41080217 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30848700 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303œâgÛã&ò ÝM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚159095987377664 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149244733690880 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _runtime‚1652.989840262 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2852097519104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28257789 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17851374 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚71928915 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159095882309120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚154220457355232 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71939074 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2843106798630 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚327 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚159095987377664 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71939074 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28257540 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30848698 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2935101528064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156243889858560 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17851377 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2843192947216 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 % _timestamp‚1.7845034989264762e+09 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156243889858560 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚156243872819200 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152179835218944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚30848698 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41080217 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30848700 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚246 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149244708787200Ç_ý 2ó - _wandbruntime‚1652  _runtime‚1652^åRô þ 2loss/SRqqt2/assignment_loss‚0.7313088965501862 .loss/FBt1/detection_loss‚2.018236989772413 epoch‚5 _step‚328 .loss/FRt1/assignment_loss‚2.67845928168797 1loss/FBt1/assignment_loss‚0.18208851922055375 1loss/SRqqt1/detection_loss‚1.3592377344580988 1loss/SRqqt2/detection_loss‚0.6717632087077253 0loss/FRt2/assignment_loss‚1.2069981851628129 2loss/SRqqt1/assignment_loss‚0.9158133510376016 0loss/FBt2/assignment_loss‚0.0814812284459529 /loss/FBt2/detection_loss‚0.7344144419287605 _runtime‚1652.991280834 /loss/FRt1/detection_loss‚0.6341058378387894 /loss/FRt2/detection_loss‚1.4300224216565536 &loss/total_loss‚12.643930096467418 trainer/global_step‚4049 % _timestamp‚1.7845034989282758e+09x0Üš õ ƒ 2loss/SRqqt2/assignment_loss‚0.7313088965501862 .loss/FBt1/detection_loss‚2.018236989772413 epoch‚5 _step‚328 .loss/FRt1/assignment_loss‚2.67845928168797 1loss/FBt1/assignment_loss‚0.18208851922055375 1loss/SRqqt1/detection_loss‚1.3592377344580988 1loss/SRqqt2/detection_loss‚0.6717632087077253 0loss/FRt2/assignment_loss‚1.2069981851628129 2loss/SRqqt1/assignment_loss‚0.9158133510376016 0loss/FBt2/assignment_loss‚0.0814812284459529 /loss/FBt2/detection_loss‚0.7344144419287605 _runtime‚1652.991280834 /loss/FRt1/detection_loss‚0.6341058378387894 /loss/FRt2/detection_loss‚1.4300224216565536 &loss/total_loss‚12.643930096467418 trainer/global_step‚4049 % _timestamp‚1.7845034989282758e+09ÈLɯ=pö ‚(:å Ö¹õÒèåÝ¿ network.sent‚ 5.229754e+06 network.recv‚502110! disk.nvme1c1n1.in‚ 10.48828125 disk.nvme1n1.in‚ 10.48828125 disk.dm-0.in‚ 98.12109375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 87.6328125( disk.nvme0c0n1.out‚12581.92236328125% disk./.usageGB‚1167.9613876342773% disk.nvme1c1n1.out‚2573.494140625$ memory_percent‚98.80731105804443( proc.memory.availableMB‚ 195.41015625* proc.memory.percent‚17.537236213684082 cpu‚26.537037637041852 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚1205.861328125) disk./.usagePercent‚76.07629947137453 disk.md0.in‚0 disk.md0.out‚ 44.69140625 disk.rbd0.in‚0& disk.nvme0n1.out‚12581.92236328125# proc.memory.rssMB‚ 2873.30078125# disk.nvme1n1.out‚2573.494140625# disk.dm-0.out‚13853.69580078125 disk.nvme0n1.in‚ 87.6328125‚{ò®š÷ ‚(: Ö¹õÒÅÕðÀ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚166.216' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚55.40533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê„†42ø - _wandbruntime‚1672  _runtime‚1672•Î”Éù à % _timestamp‚1.7845035186697915e+09 _runtime‚1672.732938513 _step‚329 ' lr-AdamW/pg1‚0.00011018492049004992 ' lr-AdamW/pg2‚0.00011018492049004992 trainer/global_step‚4099ìUÖÎú È _step‚329 ' lr-AdamW/pg1‚0.00011018492049004992 ' lr-AdamW/pg2‚0.00011018492049004992 trainer/global_step‚4099 % _timestamp‚1.7845035186697915e+09 _runtime‚1672.732938513ɾó~2û - _wandbruntime‚1672  _runtime‚1672Ò»Eü ÎO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4099 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31207427 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚158903535146540 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚156027188675472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚158074824813568 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBïÎM DeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2876260322594 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18061587 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72784643 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 _step‚330 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚160960198996992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18061584 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2885357144064 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41577214 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10526878 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚153962941610496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 % _timestamp‚1.7845035186708405e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚150993796396032 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160960093935616 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41567071 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10527120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31207429 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158074824813568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2969354725888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _runtime‚1672.737653323 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31207427 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158074841852928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2885269122048 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160960093935616bÌ`gÙ'ý ÓO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚160960093935616 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41567071 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10527120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31207429 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158074824813568 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2969354725888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _runtime‚1672.737653323 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31207427 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚158074841852928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2885269122048 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚160960093935616 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4099 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31207427 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚158903535146540 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚156027188675472 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚158074824813568 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2876260322594 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18061587 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚72784643 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 _step‚330 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚160960198996992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18061584 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2885357144064 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚41577214 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10526878 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚153962941610496 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 % _timestamp‚1.7845035186708405e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚150993796396032 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330ÊòaÔ³2þ - _wandbruntime‚1673  _runtime‚1673Ž5½®Ý&ÿ ×M SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72791717 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72791717 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41587126 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚153999539343360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚156064043703872 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2877022422624 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚158112191180800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10529253 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72801876 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚151029466649600 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18065874 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28595127 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 _step‚331 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72801876 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2969865286656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚160998243481088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚153999331936256 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41576969 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41587126 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31214750 E›pJf€ÜM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18065874 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28595127 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 _step‚331 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72801876 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2969865286656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚160998243481088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚153999331936256 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41576969 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_p‹Y&X|ool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41587126 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31214750 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72801876 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚151029466649600 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0Ë‚áV32- _wandbruntime‚1673  _runtime‚1673[Br‚ÿ 2loss/SRqqt1/assignment_loss‚0.8722357860989058 2loss/SRqqt2/assignment_loss‚0.6758897847913075 1loss/SRqqt2/detection_loss‚0.7181960586657026 .loss/FBt2/detection_loss‚5.800062754755523 epoch‚5 $ _timestamp‚1.784503518944372e+09 0loss/FRt1/assignment_loss‚2.7383289233925416 0loss/FBt1/assignment_loss‚0.2517829868747299 /loss/FRt1/detection_loss‚0.5979342931685921 /loss/FRt2/detection_loss‚1.4844045464184512 /loss/FRt2/assignment_loss‚1.099360554951888 2loss/FBt2/assignment_loss‚0.029783283984154026 1loss/SRqqt1/detection_loss‚1.3326870443137444 _runtime‚1673.007361452 _step‚332 /loss/FBt1/detection_loss‚1.9338020987343043 &loss/total_loss‚17.534468116149842 trainer/global_step‚4099ÑT6 ƒ„ /loss/FBt1/detection_loss‚1.9338020987343043 &loss/total_loss‚17.534468116149842 trainer/global_step‚4099 2loss/SRqqt1/assignment_loss‚0.8722357860989058 2loss/SRqqt2/assignment_loss‚0.6758897847913075 1loss/SRqqt2/detection_loss‚0.7181960586657026 .loss/FBt2/detection_loss‚5.800062754755523 epoch‚5 $ _timestamp‚1.784503518944372e+09 0loss/FRt1/assignment_loss‚2.7383289233925416 0loss/FBt1/assignment_loss‚0.2517829868747299 /loss/FRt1/detection_loss‚0.5979342931685921 /loss/FRt2/detection_loss‚1.4844045464184512 /loss/FRt2/assignment_loss‚1.099360554951888 2loss/FBt2/assignment_loss‚0.029783283984154026 1loss/SRqqt1/detection_loss‚1.3326870443137444 _runtime‚1673.007361452 _step‚332ÌGˆò›„‚(: å¹õÒ‚¯±¿ gpu.0.gpu‚47.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚179.294' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚59.76466666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`I¬k…‚(:à å¹õÒæÏ÷¿) disk./.usagePercent‚76.09966919760308 disk.md0.in‚0 disk.dm-0.in‚ 98.12109375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 87.6328125( disk.nvme0c0n1.out‚12934.17431640625 disk.nvme0n1.in‚ 87.6328125' proc.memory.availableMB‚ 195.2109375 network.sent‚ 5.293244e+06% disk./.usageGB‚1168.3201713562012 disk.nvme1n1.in‚ 10.48828125$ disk.nvme1n1.out‚2720.2001953125" disk.dm-1.out‚1206.15087890625# proc.memory.rssMB‚ 2873.30078125 proc.cpu.threads‚17 network.recv‚506181& disk.nvme1c1n1.out‚2720.2001953125& disk.nvme0n1.out‚12934.17431640625* proc.memory.percent‚17.537236213684082! disk.nvme1c1n1.in‚ 10.48828125 disk.md0.out‚ 45.1171875 disk.dm-0.out‚ 14351.4375 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚98.80852699279785 cpu‚26.543048724377098Gpd2†- _wandbruntime‚1693  _runtime‚1693\éxtɇà ' lr-AdamW/pg1‚0.00011012594250017946 ' lr-AdamW/pg2‚0.00011012594250017946 trainer/global_step‚4149 % _timestamp‚1.7845035392177868e+09 _runtime‚1693.281553668 _step‚333ó|ØÎˆÈ ' lr-AdamW/pg1‚0.00011012594250017946 ' lr-AdamW/pg2‚0.00011012594250017946 trainer/global_step‚4149 % _timestamp‚1.7845035392177868e+09 _runtime‚1693.281553668 _step‚333Í¢­ýß2‰- _wandbruntime‚1693  _runtime‚1693xójÓ'ŠÍO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚159943143175168 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _runtime‚1693.281840949 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚157870792063472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚162862350039040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2919294885888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3004119636992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 trainer/global_step‚4149 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42073966 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194831360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73647445 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31573477 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160780968009948 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73637300 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚162862455100416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2919206863872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚160780864822114 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚159943160214528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18276084 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42063823 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚162862455100416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚159943160214528 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚155782843823104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18276087 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3004289564672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31573479 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚155782648991744 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2919294885888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 $ _timestamp‚1.784503539218545e+09 _step‚334 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208€p`Ø'‹ÒO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157870775024112 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚73647445 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31573477 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160780968009948 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚73637300 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚162862455100416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2919206863872 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚160780864822114 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚159943160214528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18276084 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42063823 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚162862455100416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚159943160214528 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚155782843823104 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18276087 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3004289564672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31573479 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚155782648991744 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2919294885888 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 $ _timestamp‚1.784503539218545e+09 _step‚334 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚159943143175168 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 _runtime‚1693.281840949 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚157870792063472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚162862350039040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2919294885888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3004119636992 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 trainer/global_step‚4149 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42073966 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194831360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 I4V2Œ- _wandbruntime‚1693  _runtime‚1693ßÿ½Â ×M O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2919884973568 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31580798 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73664678 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3005000409600 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42083878 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31580800 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚157907664131232 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2910765749446 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚162900499584512 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚159980509542400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10652317 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚152814249415680 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚159980526581760 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73664678 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚162900394515968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚1693.548035293 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚159980526581760 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10652560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚160818516029264 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2919973002752 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42083878 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚160818412841318 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73654519 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚152814224512000 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚162900499584512 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31580798 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18280374 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28932691 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586\Ôžâ&ŽÜM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 trainer/global_step‚4149 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42083878 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31580800 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚157907664131232 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2910765749446 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚162900499584512 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚159980509542400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10652317 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚152814249415680 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚159980526581760 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚73664678 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚162900394515968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _runtime‚1693.548035293 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚159980526581760 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10652560 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚160818516029264 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2919973002752 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚42083878 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚160818412841318 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚73654519 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚152814224512000 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚162900499584512 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31580798 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18280374 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28932691 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2919884973568 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31580798 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚73664678 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3005000409600 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 Bþ /loss/FRt2/detection_loss‚1.4895475427557667 1loss/SRqqt2/detection_loss‚0.7003750141541925 /loss/FBt1/detection_loss‚0.9744289029816476 &loss/total_loss‚11.779201268931734 trainer/global_step‚4149 1loss/SRqqt1/detection_loss‚1.2973531127717621 _step‚336 2loss/SRqqt1/assignment_loss‚0.8511303937097114 1loss/FBt1/assignment_loss‚0.23261115276751335 _runtime‚1693.548591214 1loss/SRqqt2/assignment_loss‚0.718345875338782 1loss/FBt2/assignment_loss‚0.19648370303159132 /loss/FRt1/detection_loss‚0.5680170861185009 /loss/FBt2/detection_loss‚0.8751064154169729 $ _timestamp‚1.784503539485709e+09 epoch‚5 /loss/FRt1/assignment_loss‚2.684584732353687 /loss/FRt2/assignment_loss‚1.191217337531605—[ëC ‘ƒ 1loss/FBt1/assignment_loss‚0.23261115276751335 _runtime‚1693.548591214 1loss/SRqqt2/assignment_loss‚0.718345875338782 1loss/FBt2/assignment_loss‚0.19648370303159132 /loss/FRt1/detection_loss‚0.5680170861185009 /loss/FBt2/detection_loss‚0.8751064154169729 $ _timestamp‚1.784503539485709e+09 epoch‚5 /loss/FRt1/assignment_loss‚2.684584732353687 /loss/FRt2/assignment_loss‚1.191217337531605 /loss/FRt2/detection_loss‚1.4895475427557667 1loss/SRqqt2/detection_loss‚0.7003750141541925 /loss/FBt1/detection_loss‚0.9744289029816476 &loss/total_loss‚11.779201268931734 trainer/global_step‚4149 1loss/SRqqt1/detection_loss‚1.2973531127717621 _step‚336 2loss/SRqqt1/assignment_loss‚0.8511303937097114Ђ²Ÿön’‚(:ã ô¹õÒ ‘Ì¿ disk.dm-0.out‚ 14357.171875( proc.memory.availableMB‚ 195.21484375 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 87.6328125 disk.nvme1n1.in‚ 10.48828125% disk.nvme1n1.out‚2721.05029296875 disk.md0.in‚0 disk.md0.out‚ 45.5390625 cpu‚26.548825816611735) disk./.usagePercent‚76.09971317750316% disk./.usageGB‚1168.3208465576172" disk.dm-1.out‚1206.26806640625! disk.nvme1c1n1.in‚ 10.48828125' disk.nvme1c1n1.out‚2721.05029296875# proc.memory.rssMB‚ 2873.30078125 proc.cpu.threads‚17 network.sent‚ 5.354585e+06 network.recv‚510677 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 87.6328125 disk.dm-0.in‚ 98.12109375$ memory_percent‚98.80850315093994* proc.memory.percent‚17.537236213684082' disk.nvme0c0n1.out‚12940.0634765625% disk.nvme0n1.out‚12940.0634765625 » ›“‚(: ô¹õÒ„ñü¿ gpu.0.gpu‚43.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚190.985' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚63.66166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[i¬l”‚(:á ƒºõÒÁëú¿! disk.nvme1c1n1.in‚ 10.48828125 disk.md0.out‚ 45.7265625# proc.memory.rssMB‚ 2873.30078125 proc.cpu.threads‚17% disk.nvme1c1n1.out‚2724.560546875 disk.nvme1n1.in‚ 10.48828125 disk.md0.in‚0! disk.dm-1.out‚1206.5166015625 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚12978.50732421875& disk.nvme0n1.out‚12978.50732421875 network.sent‚ 5.41691e+06 network.recv‚514817( disk./.usagePercent‚76.1032516959047# disk.nvme1n1.out‚2724.560546875# disk.dm-0.out‚14398.45556640625 disk.nvme0c0n1.in‚ 87.6328125 disk.nvme0n1.in‚ 87.6328125# memory_percent‚98.8072156906128$ disk./.usageGB‚1168.375171661377 disk.dm-0.in‚ 98.12109375 disk.dm-1.in‚0( proc.memory.availableMB‚ 195.42578125* proc.memory.percent‚17.537236213684082 cpu‚26.554189408391377£EÛ•‚(: ƒºõÒ¿í¸À gpu.0.gpu‚31.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.367' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.12233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³P2–- _wandbruntime‚1713  _runtime‚1713‰9]»É—à ' lr-AdamW/pg2‚0.00011006609859068987 trainer/global_step‚4199 % _timestamp‚1.7845035596428218e+09 _runtime‚1713.705955303 _step‚337 ' lr-AdamW/pg1‚0.00011006609859068987gɧñÎ˜È ' lr-AdamW/pg1‚0.00011006609859068987 ' lr-AdamW/pg2‚0.00011006609859068987 trainer/global_step‚4199 % _timestamp‚1.7845035596428218e+09 _runtime‚1713.705955303 _step‚337ÑVüƒ¼2™- _wandbruntime‚1713  _runtime‚1713Õt±Ô'šÎO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2953144605696 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚162658400873356 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164764711203840 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚159714395451472 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2943919273410 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2953144605696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3039232799744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _step‚338 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2953232627712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚161811478576128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2953232627712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2944005421884 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31939527 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚161811461536768 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29263869 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164764606142464 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74510247 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31939529 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18490587 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚157602331214848 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3039043997696 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚161811461536768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164764606142464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42560575 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚159714378412112 trainer/global_step‚4199 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31939529 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 % _timestamp‚1.7845035596435223e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42570718 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚164764711203840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚162658297685522 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0fˆVL½›ÓO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚164764711203840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚162658297685522 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2953144605696 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚162658400873356 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚164764711203840 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚159714395451472 Z»ñ´l$HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2943919273410 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2953144605696 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3039232799744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _step‚338 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2953232627712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚161811478576128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2953232627712 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2944005421884 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚31939527 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚161811461536768 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29263869 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚164764606142464 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚74510247 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31939529 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18490587 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚157602331214848 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3039043997696 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚161811461536768 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚164764606142464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42560575 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚159714378412112 trainer/global_step‚4199 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31939529 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 % _timestamp‚1.7845035596435223e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42570718 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0ÒÊe(c2œ- _wandbruntime‚1713  _runtime‚1713†‡jÝ&×M OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31946848 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2953822715392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74517321 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31946850 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2953822715392 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18494874 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2953910744576 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚157638731188736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3039748814336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚4199 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚159751267519232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42570473 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18494877 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164802755687936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2944595224854 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3039922929152 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31946848 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚161848827904000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29270385 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚159751250479872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚154599007278080 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74527480 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42580630 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74527480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚154598982374400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 $ _timestamp‚1.784503559916353e+09 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2944681373440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42570473 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10775769 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚164802650619392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚162695948892672=DŸ–â&žÜM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2944681373440 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚42570473 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10775769 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚164802650619392 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚162695948892672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31946848 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2953822715392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚74517321 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚31946850 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2953822715392 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18494874 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2953910744576 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚157638731188736 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3039748814336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚4199 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚159751267519232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42570473 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18494877 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚164802755687936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2944595224854 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3039922929152 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚31946848 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚161848827904000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29270385 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚159751250479872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚154599007278080 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74527480 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚42580630 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74527480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚154598982374400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 $ _timestamp‚1.784503559916353e+09Ó¾±)2Ÿ- _wandbruntime‚1713  _runtime‚1713WOC  /loss/FRt1/assignment_loss‚2.650264074210248 0loss/FRt2/assignment_loss‚1.1917932947476706 1loss/FBt1/assignment_loss‚0.43947724957170065 &loss/total_loss‚11.868986054884763 trainer/global_step‚4199 /loss/FRt1/detection_loss‚0.5653566481519063 /loss/FRt2/detection_loss‚1.4253740895875833 _runtime‚1713.980243708 2loss/SRqqt2/assignment_loss‚0.6980880655785751 1loss/SRqqt2/detection_loss‚0.6968831390046455 /loss/FBt2/detection_loss‚0.7971820107751539 % _timestamp‚1.7845035599174116e+09 _step‚340 2loss/SRqqt1/assignment_loss‚0.9139300968746344 1loss/FBt2/assignment_loss‚0.07069919380530587 1loss/SRqqt1/detection_loss‚1.2762513128916422 /loss/FBt1/detection_loss‚1.1436868796856963 epoch‚5RßÔa ¡† 2loss/SRqqt2/assignment_loss‚0.6980880655785751 1loss/SRqqt2/detection_loss‚0.6968831390046455 /loss/FBt2/detection_loss‚0.7971820107751539 % _timestamp‚1.7845035599174116e+09 _step‚340 2loss/SRqqt1/assignment_loss‚0.9139300968746344 1loss/FBt2/assignment_loss‚0.07069919380530587 1loss/SRqqt1/detection_loss‚1.2762513128916422 /loss/FBt1/detection_loss‚1.1436868796856963 epoch‚5 /loss/FRt1/assignment_loss‚2.650264074210248 0loss/FRt2/assignment_loss‚1.1917932947476706 1loss/FBt1/assignment_loss‚0.43947724957170065 &loss/total_loss‚11.868986054884763 trainer/global_step‚4199 /loss/FRt1/detection_loss‚0.5653566481519063 /loss/FRt2/detection_loss‚1.4253740895875833 _runtime‚1713.980243708ÔCϤsg¢‚(:Ü ’ºõÒïìã¿( proc.memory.availableMB‚ 195.39453125* proc.memory.percent‚17.537236213684082 cpu‚26.560340542566156 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 87.6328125! disk.nvme1c1n1.in‚ 10.48828125& disk.nvme1c1n1.out‚2784.5537109375 disk.md0.in‚0) disk./.usagePercent‚76.11519956876043% disk./.usageGB‚1168.5586013793945 disk.dm-1.in‚0! disk.dm-1.out‚1206.6806640625# disk.nvme0c0n1.out‚ 13165.984375 disk.nvme1n1.in‚ 10.48828125 disk.md0.out‚ 46.16796875$ memory_percent‚98.80740642547607 network.sent‚ 5.421768e+06! disk.dm-0.out‚14644.830078125 disk.rbd0.out‚0! disk.nvme0n1.out‚ 13165.984375$ disk.nvme1n1.out‚2784.5537109375# proc.memory.rssMB‚ 2873.30078125 proc.cpu.threads‚17 network.recv‚518399 disk.dm-0.in‚ 98.12109375 disk.nvme0n1.in‚ 87.6328125/õÅ6œ£‚(:‘ ’ºõÒϨÀ gpu.0.gpu‚46.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚189.965' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚63.321666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý:¬2¤- _wandbruntime‚1734  _runtime‚1734™pkÐȥ $ _timestamp‚1.784503580407122e+09 _runtime‚1734.470262174 _step‚341 ' lr-AdamW/pg1‚0.00011000538972010286 ' lr-AdamW/pg2‚0.00011000538972010286 trainer/global_step‚4249@¯¸:Í¦Ç $ _timestamp‚1.784503580407122e+09 _runtime‚1734.470262174 _step‚341 ' lr-AdamW/pg1‚0.00011000538972010286 ' lr-AdamW/pg2‚0.00011000538972010286 trainer/global_step‚4249Õ©M5S2§- _wandbruntime‚1734  _runtime‚17343 {ÌP¨ÎO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚1óB„'52 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32305577 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32305579 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2987082347520 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29600984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10896132 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29601219 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3074164178944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚163679779898368 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75373049 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32305579 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _runtime‚1734.471294706 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75362904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18705084 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚166666967307264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚156348045079552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚164535730548930 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2987170369536 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚161557998839472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2987082347520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2977834897292 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 trainer/global_step‚4249 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3073975376896 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚161557981800112 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚163679796937728 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43057327 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚159422234162176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚164535833736764 % _timestamp‚1.7845035804081633e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32305577 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166666862245888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚159422020456448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0[‚ ùÙ'©ÓO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18705084 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚166666967307264 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚156348045079552 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚164535730548930 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2987170369536 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚161557998839472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2987082347520 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2977834897292 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 trainer/global_step‚4249 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3073975376896 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚161557981800112 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚163679796937728 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43057327 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚159422234162176 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚164535833736764 % _timestamp‚1.7845035804081633e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32305577 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166666862245888 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚159422020456448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32305577 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32305579 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2987082347520 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29600984 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10896132 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29601219 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3074164178944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚163679779898368 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75373049 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32305579 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _runtime‚1734.471294706 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75362904 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448Ö+¹y-2ª- _wandbruntime‚1734  _runtime‚1734íÕÙÞ&«ØM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75380123 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43077382 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 trainer/global_step‚4249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159458623156736 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2987760457216 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚163717163304960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32312900 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 _runtime‚1734.744933215 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29607693 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _step‚343 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2987760457216 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚156383765140480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚161594853867872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2987848486400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75390282 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚164573381756080 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32312898 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75390282 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚163717163304960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2978510848848 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚164573278568134 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32312900 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚166704906722816 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 % _timestamp‚1.7845035806816182e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18709374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43077382 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚166705011791360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3074675512832 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚163717146265600 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚163717146265600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2978424700262 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43067225 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚161594870907232 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29607933 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10898319 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚156383740236800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚166705011791360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104]Doáp ¬ÝM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚166704906722816 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 % _timestamp‚1.7845035806816182e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18709374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43077382 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚166705011791360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3074675512832 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚163717146265600 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚163717146265600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2978424700262 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43067225 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚161594870907232 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29607933 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10898319 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚156383740236800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/act‹E?sive_bytes.all.allocated‚166705011791360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75380123 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43077382 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 trainer/global_step‚4249 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159458623156736 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2987760457216 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚163717163304960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32312900 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 _runtime‚1734.744933215 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29607693 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _step‚343 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2987760457216 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚156383765140480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚161594853867872 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2987848486400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75390282 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚164573381756080 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32312898 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚75390282 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚163717163304960 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2978510848848 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚164573278568134 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32312900×Ðx72­- _wandbruntime‚1734  _runtime‚1734Nå l®ü % _timestamp‚1.7845035806826534e+09 1loss/SRqqt1/detection_loss‚1.3621996431829757 /loss/FBt2/detection_loss‚1.1922050545730916 1loss/SRqqt1/assignment_loss‚0.748255099839298 2loss/SRqqt2/assignment_loss‚0.7374277447825992 1loss/FBt1/assignment_loss‚0.49025445322918887 /loss/FRt2/detection_loss‚1.4837014488275948 -loss/FBt1/detection_loss‚0.92567185990697 &loss/total_loss‚12.441404061866663 epoch‚5 _runtime‚1734.745486516 /loss/FRt1/assignment_loss‚2.762821813257117 /loss/FRt2/assignment_loss‚1.272881120329159 0loss/FBt2/assignment_loss‚0.1132720230436285 _step‚344 /loss/FRt1/detection_loss‚0.5586038797840133 1loss/SRqqt2/detection_loss‚0.7941099211110274 trainer/global_step‚4249Ÿm”¯ /loss/FRt1/assignment_loss‚2.762821813257117 /loss/FRt2/assignment_loss‚1.272881120329159 0loss/FBt2/assignment_loss‚0.1132720230436285 _step‚344 /loss/FRt1/detection_loss‚0.5586038797840133 1loss/SRqqt2/detection_loss‚0.7941099211110274 trainer/global_step‚4249 % _timestamp‚1.7845035806826534e+09 1loss/SRqqt1/detection_loss‚1.3621996431829757 /loss/FBt2/detection_loss‚1.1922050545730916 1loss/SRqqt1/assignment_loss‚0.748255099839298 2loss/SRqqt2/assignment_loss‚0.7374277447825992 1loss/FBt1/assignment_loss‚0.49025445322918887 /loss/FRt2/detection_loss‚1.4837014488275948 -loss/FBt1/detection_loss‚0.92567185990697 &loss/total_loss‚12.441404061866663 epoch‚5 _runtime‚1734.745486516ØA8Ìj°‚(:ß ¡ºõÒÌ—¾¿! disk.dm-1.out‚1206.8427734375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚13177.86962890625& disk.nvme0n1.out‚13177.86962890625$ memory_percent‚98.80778789520264( proc.memory.availableMB‚ 195.33203125# disk.dm-0.out‚14658.04052734375 disk.dm-1.in‚0 network.sent‚ 5.484334e+06 network.recv‚522735! disk.nvme1c1n1.in‚ 10.48828125# disk.nvme1c1n1.out‚ 2786.5390625 disk.nvme1n1.in‚ 10.48828125! disk.nvme1n1.out‚ 2786.5390625% disk./.usageGB‚1168.5585823059082 disk.nvme0c0n1.in‚ 87.6328125 disk.nvme0n1.in‚ 87.6328125* proc.memory.percent‚17.537236213684082 proc.cpu.threads‚17) disk./.usagePercent‚76.11519832639037 disk.md0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2873.30078125 cpu‚26.56567739601992 disk.md0.out‚ 46.390625 disk.dm-0.in‚ 98.12109375!ö¹š±‚(: ¡ºõÒ¯‚À gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚199.034' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.34466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªF´2²- _wandbruntime‚1754  _runtime‚1754›í¬Oɳà _step‚345 ' lr-AdamW/pg1‚0.00010994381686079419 ' lr-AdamW/pg2‚0.00010994381686079419 trainer/global_step‚4299 % _timestamp‚1.7845036007898147e+09 _runtime‚1754.852916826ÐǽSÎ´È % _timestamp‚1.7845036007898147e+09 _runtime‚1754.852916826 _step‚345 ' lr-AdamW/pg1‚0.00010994381686079419 ' lr-AdamW/pg2‚0.00010994381686079419 trainer/global_step‚4299Ùók,®2µ- _wandbruntime‚1754  _runtime‚17541kŸÔ'¶ÎO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76225706 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32671627 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29938235 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18919584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚158132827845632 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3021108111360 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3021108111360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _step‚346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29938476 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚163401585188112 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32671627 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168569118349312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3021020089344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚158132802941952 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166413266600172 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18919587 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32671629 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43554079 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166413163412338 % _timestamp‚1.7845036007908206e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168569223410688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32671629 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4299 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161241965155840 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3011578224226 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43564222 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43554079 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚165548098259968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚165548098259968 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168569118349312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11018651 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚1754.853954135 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161241761935872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚163401602227472 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚165548115299328 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43564222ŸnÙ'·ÓO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161241965155840 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3011578224226 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43564222 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43554079 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚165548098259968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚165548098259968 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168569118349312 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11018651 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚1754.853954135 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161241761935872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚163401602227472 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚165548115299328 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚43564222 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚76225706 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32671627 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚29938235 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18919584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚158132827845632 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3021108111360 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3021108111360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _step‚346 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29938476 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚163401585188112 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32671627 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168569118349312 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3021020089344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚158132802941952 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚166413266600172 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18919587 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32671629 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚43554079 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚166413163412338 % _timestamp‚1.7845036007908206e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168569223410688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32671629 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚4299ÚÂcÆío¸‚(:ä °ºõÒ•Ùî¿ disk.nvme0n1.in‚ 87.6328125$ disk.nvme1n1.out‚2787.1669921875 disk.md0.in‚0 disk.md0.out‚ 46.828125# disk.dm-0.out‚14658.59521484375 cpu‚26.571185346839794 network.sent‚ 5.547953e+06 network.recv‚527260) disk./.usagePercent‚76.11520925924688 disk.nvme0c0n1.in‚ 87.6328125( disk.nvme0c0n1.out‚13178.81396484375# proc.memory.rssMB‚ 2873.30078125$ disk./.usageGB‚1168.558750152588! disk.dm-1.out‚1206.9326171875 disk.rbd0.out‚0& disk.nvme0n1.out‚13178.81396484375! disk.nvme1c1n1.in‚ 10.48828125 disk.dm-0.in‚ 98.12109375' proc.memory.availableMB‚ 195.0859375 proc.cpu.threads‚17 disk.rbd0.in‚0& disk.nvme1c1n1.out‚2787.1669921875 disk.nvme1n1.in‚ 10.48828125$ memory_percent‚98.80928993225098* proc.memory.percent‚17.537236213684082 disk.dm-1.in‚0£¡ nœ¹‚(:‘ °ºõÒ¦²ÄÀ gpu.0.gpu‚69.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚155.464' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚51.82133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆÌÁ2º- _wandbruntime‚1755  _runtime‚1755 ¸ô=Ý&»×M REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32678950 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29945288 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3109657232384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 _runtime‚1755.126583946 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚163438457255872 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚165585481666560 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚163438474295232 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32678950 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76253084 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3012254175670 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚168607267894784 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3109843930112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3021698199040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚158168498099200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18923874 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11021161 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚347 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43563977 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76242925 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29945035 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32678948 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76253084 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32678948 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11021411 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.784503601063293e+09¸€Tâ&¼ÜM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3012340324256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚76242925 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3021786228224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43563977 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚166450814619488 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚76242925 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29945035 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32678948 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚76253084 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32678948 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11021411 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 $ _timestamp‚1.784503601063293e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32678950 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29945288 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3109657232384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 _runtime‚1755.126583946 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚163438457255872 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚165585481666560 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚163438474295232 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32678950 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚76253084 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3012254175670 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚168607267894784 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3109843930112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3021698199040 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚158168498099200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18923874 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11021161 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚347 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43563977 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159917585708032 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3055045853184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19134084 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77088508 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚167416433660928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚167416416621568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44050831 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3045407699634 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3144065159168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚168290596275746 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚165245205615472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44050831 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7845036215369306e+09 _step‚350 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚170471479514112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚165245188576112 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170471374452736 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159917560804352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33037679 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30275950 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19134087 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚170471374452736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚163061650867200 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77098653 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚163061456035840 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11141866 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3045493848108 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚167416433660928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33037677 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 _runtime‚1775.602598076 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44060974 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11142115 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3143895231488 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44060974 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚170471479514112 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3054957831168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3055045853184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 trainer/global_step‚4349 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33037679 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚123035EdÙ'ÇÓO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19134087 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚170471374452736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚163061650867200 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77098653 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚163061456035840 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11141866 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3045493848108 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚167416433660928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33037677 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 _runtime‚1775.602598076 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44060974 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11142115 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3143895231488 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44060974 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚170471479514112 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3054957831168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3055045853184 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 trainer/global_step‚4349 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33037679 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3054957831168 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159917585708032 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3055045853184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19134084 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77088508 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚167416433660928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚167416416621568 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44050831 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3045407699634 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3144065159168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚168290596275746 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚165245205615472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44050831 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 % _timestamp‚1.7845036215369306e+09 _step‚350 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚170471479514112 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚165245188576112 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚170471374452736 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159917560804352 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33037679 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30275950 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474Þ¯î/i2È- _wandbruntime‚1775  _runtime‚17751c}tÝ&É×M PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚167453800028160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 _runtime‚1775.874219435 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚167453782988800 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚168328247482896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚163098052597248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3144587131392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33045000 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784503621810938e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚170509523998208 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚170509523998208 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3055635940864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚167453782988800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3055635940864 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77115886 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33044998 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11144363 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44060729 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11144622 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30282737 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3046169799664 trainer/global_step‚4349 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3144771731968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44060729 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3046083651078 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19138374 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77105727 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3055723970048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33045000 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3055723970048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚170509418929664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44070886 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30282999 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0E~çjâ&ÊÜM NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44060729 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11144622 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30282737 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3046169799664 trainer/global_step‚4349 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3144771731968 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44060729 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3046083651078 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19138374 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77105727 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3055723970048 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33045000 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3055723970048 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚170509418929664 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44070886 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30282999 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚167453800028160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 _runtime‚1775.874219435 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚167453782988800 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚168328247482896 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚163098052597248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3144587131392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33045000 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784503621810938e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚170509523998208 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚170509523998208 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3055635940864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚167453782988800 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3055635940864 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77115886 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33044998 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11144363 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0ßÛ™¶#2Ë- _wandbruntime‚1775  _runtime‚1775zGƒ6Ì‚ /loss/FBt2/detection_loss‚0.7452106744420486 &loss/total_loss‚12.536191390022086 % _timestamp‚1.7845036218119867e+09 /loss/FRt1/assignment_loss‚2.709095014859054 0loss/FRt2/assignment_loss‚1.1961987635668587 epoch‚5 trainer/global_step‚4349 _step‚352 1loss/SRqqt1/detection_loss‚1.3160792088698752 _runtime‚1775.874835286 1loss/FBt1/assignment_loss‚0.18001839482723667 /loss/FRt1/detection_loss‚0.5888403399559616 /loss/FRt2/detection_loss‚1.7071292272526954 2loss/SRqqt1/assignment_loss‚1.0906391160743725 2loss/SRqqt2/assignment_loss‚0.7804557656492173 2loss/FBt2/assignment_loss‚0.047367236059890835 1loss/SRqqt2/detection_loss‚0.7554089122481356 /loss/FBt1/detection_loss‚1.4197487362167414ZW×Ý Í‡ 1loss/SRqqt2/detection_loss‚0.7554089122481356 /loss/FBt1/detection_loss‚1.4197487362167414 /loss/FBt2/detection_loss‚0.7452106744420486 &loss/total_loss‚12.536191390022086 % _timestamp‚1.7845036218119867e+09 /loss/FRt1/assignment_loss‚2.709095014859054 0loss/FRt2/assignment_loss‚1.1961987635668587 epoch‚5 trainer/global_step‚4349 _step‚352 1loss/SRqqt1/detection_loss‚1.3160792088698752 _runtime‚1775.874835286 1loss/FBt1/assignment_loss‚0.18001839482723667 /loss/FRt1/detection_loss‚0.5888403399559616 /loss/FRt2/detection_loss‚1.7071292272526954 2loss/SRqqt1/assignment_loss‚1.0906391160743725 2loss/SRqqt2/assignment_loss‚0.7804557656492173 2loss/FBt2/assignment_loss‚0.047367236059890835à=£€Î‚(:Þ κõÒˆÄÔ¿% disk.nvme1c1n1.out‚2788.521484375 disk.dm-0.in‚ 98.12109375 disk.dm-1.in‚0 disk.rbd0.Odòìéout‚0( proc.memory.availableMB‚ 195.26171875* proc.memory.percent‚17.537236213684082 proc.cpu.threads‚17 network.sent‚ 5.650689e+06) disk./.usagePercent‚76.11520453824066 disk.nvme1n1.in‚ 10.48828125# disk.nvme1n1.out‚2788.521484375 disk.md0.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚13204.8818359375$ memory_percent‚98.80821704864502 disk.nvme0n1.in‚ 87.6328125! disk.nvme1c1n1.in‚ 10.48828125 disk.md0.out‚47.53125# proc.memory.rssMB‚ 2873.30078125 network.recv‚535642% disk.nvme0n1.out‚13204.8818359375" disk.dm-0.out‚14684.2705078125 disk.dm-1.out‚ 1207.16015625 disk.nvme0c0n1.in‚ 87.6328125 cpu‚26.580009820366655% disk./.usageGB‚1168.5586776733398·Âþ;œÏ‚(:‘ κõÒ£Ó­À gpu.0.gpu‚46.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚133.798' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚44.599333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚b›+2Ð- _wandbruntime‚1796  _runtime‚1796SÍ×ÉÑà trainer/global_step‚4399 % _timestamp‚1.7845036423275821e+09 _runtime‚1796.394578262 _step‚353 ' lr-AdamW/pg1‚0.00010981808313469192 ' lr-AdamW/pg2‚0.00010981808313469192VQ¨ÎÒÈ ' lr-AdamW/pg2‚0.00010981808313469192 trainer/global_step‚4399 % _timestamp‚1.7845036423275821e+09 _runtime‚1796.394578262 _step‚353 ' lr-AdamW/pg1‚0.00010981808313469192áöa2Ó- _wandbruntime‚1796  _runtime‚1796+‰ Ó'ÔÍO $ _timestamp‚1.784503642328271e+09 _step‚354 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3088895572992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3179016186368 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77951310 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚169284752022528 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33403727 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77961455 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚169284752022528 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44547583 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161702343570432 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚169284734983168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44557726 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30613758 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚167088791964112 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172373630556160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33403729 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44547583 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11265174 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚170168029139154 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455^4§UØ'ÕÒO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30614020 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚44557726 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172373630556160 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33403729 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44547583 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11265174 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚170168029139154 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 $ _timestamp‚1.784503642328271e+09 _step‚354 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3088895572992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3179016186368 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚77951310 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚169284752022528 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33403727 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77961455 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚169284752022528 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚44547583 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161702343570432 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚169284734983168 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44557726 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30613758 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚167088791964112 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44567638 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚172411780101632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3179532796928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚169322118389760 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚169322101350400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3089661711872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30620740 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚167125664031872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3079913126486 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3089573682688 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77978688 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33411050 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚170205680346304 % _timestamp‚1.7845036426001706e+09 _step‚355 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3089661711872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3179719494656 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164917546620928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33411048 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30620482 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚169322118389760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3079999275072 trainer/global_step‚4399 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77968529 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚167125664031872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3079913126486 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.¥žÍ$k$peak‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3089573682688 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77978688 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33411050 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚170205680346304 % _timestamp‚1.7845036426001706e+09 _step‚355 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3089661711872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3179719494656 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164917546620928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33411048 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30620482 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚169322118389760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3079999275072 trainer/global_step‚4399 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77968529 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44567638 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚172411780101632 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3179532796928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚169322118389760 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚169322101350400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3089661711872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30620740ã $2Ù- _wandbruntime‚1796  _runtime‚1796–}–iÚý 0loss/FRt1/assignment_loss‚2.5856646930033733 1loss/FBt2/assignment_loss‚0.10541379657410778 /loss/FRt1/detection_loss‚0.5845033180648973 1loss/SRqqt1/detection_loss‚1.1913900648316427 .loss/FBt1/detection_loss‚1.496510432412227 epoch‚5 % _timestamp‚1.7845036426014056e+09 _runtime‚1796.664268742 2loss/SRqqt2/assignment_loss‚0.6219172703431468 /loss/FRt2/detection_loss‚1.4159180520216725 1loss/SRqqt2/detection_loss‚0.6584990120609291 /loss/FBt2/detection_loss‚0.9680273794003317 $loss/total_loss‚11.7603489131377 trainer/global_step‚4399 _step‚356 /loss/FRt2/assignment_loss‚1.178743272065268 2loss/SRqqt1/assignment_loss‚0.6486333375626312 0loss/FBt1/assignment_loss‚0.3051282847974714%e¼Û‚ /loss/FRt2/assignment_loss‚1.178743272065268 2loss/SRqqt1/assignment_loss‚0.6486333375626312 0loss/FBt1/assignment_loss‚0.3051282847974714 0loss/FRt1/assignment_loss‚2.5856646930033733 1loss/FBt2/assignment_loss‚0.10541379657410778 /loss/FRt1/detection_loss‚0.5845033180648973 1loss/SRqqt1/detection_loss‚1.1913900648316427 .loss/FBt1/detection_loss‚1.496510432412227 epoch‚5 % _timestamp‚1.7845036426014056e+09 _runtime‚1796.664268742 2loss/SRqqt2/assignment_loss‚0.6219172703431468 /loss/FRt2/detection_loss‚1.4159180520216725 1loss/SRqqt2/detection_loss‚0.6584990120609291 /loss/FBt2/detection_loss‚0.9680273794003317 $loss/total_loss‚11.7603489131377 trainer/global_step‚4399 _step‚356ä·6ÿ>tÜ‚(:é ݺõÒˆÔñ¿ disk.nvme1n1.in‚ 10.48828125 disk.md0.in‚0 proc.cpu.threads‚17( disk.nvme0c0n1.out‚13206.85888671875& disk.nvme0n1.out‚13206.85888671875 network.recv‚539726% disk./.usageGB‚1168.5591506958008 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 87.6328125 disk.dm-0.in‚ 98.12109375 disk.rbd0.out‚0 disk.nvme0n1.in‚ 87.6328125% disk.nvme1n1.out‚2789.31005859375# memory_percent‚98.8086462020874* proc.memory.percent‚17.537236213684082 cpu‚26.58457807162951 network.sent‚ 5.727025e+06# disk.dm-0.out‚14686.17041015625" disk.dm-1.out‚1207.27783203125 disk.rbd0.in‚0' disk.nvme1c1n1.out‚2789.31005859375 disk.md0.out‚ 47.87890625( proc.memory.availableMB‚ 195.19140625# proc.memory.rssMB‚ 2873.30078125) disk./.usagePercent‚76.11523534901814! disk.nvme1c1n1.in‚ 10.48828125,Á†ó›Ý‚(: ݺõÒÛÕžÀ gpu.0.gpu‚46.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚197.638' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.87933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MòÊìtÞ‚(:é ìºõÒèÓ¿ disk.nvme1n1.in‚ 10.48828125# disk.dm-0.out‚14693.47119140625 disk.dm-1.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚13214.43310546875 disk.md0.in‚0 disk.dm-0.in‚ 98.12109375# memory_percent‚98.8086462020874# proc.memory.rssMB‚ 2873.30078125* proc.memory.percent‚17.537236213684082 cpu‚26.589632244723756 network.sent‚ 5.774539e+06 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 87.6328125% disk.nvme1n1.out‚2789.92333984375( proc.memory.availableMB‚ 195.19140625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 87.6328125' disk.nvme1c1n1.out‚2789.92333984375 disk.md0.out‚ 48.23046875" disk.dm-1.out‚1207.38330078125 network.recv‚543524) disk./.usagePercent‚76.09975144250096$ disk./.usageGB‚1168.321434020996& disk.nvme0n1.out‚13214.43310546875! disk.nvme1c1n1.in‚ 10.48828125önlœß‚(:‘ ìºõÒ¢™ŽÀ gpu.0.gpu‚74.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚202.045' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.34833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—קq2à- _wandbruntime‚1817  _runtime‚1817Ëì›ËÈá % _timestamp‚1.7845036630039399e+09 _runtime‚ 1817.06709263 _step‚357 ' lr-AdamW/pg1‚0.00010975392428177914 ' lr-AdamW/pg2‚0.00010975392428177914 trainer/global_step‚4449•øÝÁÍâÇ trainer/global_step‚4449 % _timestamp‚1.7845036630039399e+09 _runtime‚ 1817.06709263 _step‚357 ' lr-AdamW/pg1‚0.00010975392428177914 ' lr-AdamW/pg2‚0.00010975392428177914åã >;2ã- _wandbruntime‚1817  _runtime‚1817+kúÔ'äÎO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚174275991720960 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30951145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚166701092464128 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33769777 _step‚358 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11388058 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33769779 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30950906 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚168932412391472 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78814112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33769779 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3122921336832 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174275886659584 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19563087 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11387822 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3122921336832 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45044335 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45044335 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚171153070384128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45054478 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚166700880855552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3113152798924 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚172045462002562 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 _runtime‚1817.071297487 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33769777 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3213991031296 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚163487101432832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚168932395352112 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3122833314816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45054478 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19563084 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3122833314816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚171153053344768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3113066650450 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 % _timestamp‚1.7845036630049634e+09^¸ÀŠ !åÓO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33769777 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3213991031296 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚163487101432832 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚168932395352112 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3122833314816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚45054478 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19563084 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3122833314816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚171153053344768 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3113066650450 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 % _timestamp‚1.7845036630049634e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚174275991720960 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30951145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚166701092464128 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33769777 _step‚358 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11388058 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33769779 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30950906 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚168932412391472 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚78814112 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33769779 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3122921336832 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚174275886659584 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19563087 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11387822 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3122921336832 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeÅ_ºÏviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚45044335 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45044335 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚171153070384128 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45054478 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚166700880855552 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3113152798924 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚172045462002562 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 _runtime‚1817.071297487æ9•ä2æ- _wandbruntime‚1817  _runtime‚1817Ê2Þ&çØM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚171190436751360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3123599453696 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3123511424512 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19567374 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚166737474127872 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚171190419712000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30957648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚168969267419872 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78841490 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33777100 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167823360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚174314036205056 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33777098 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78841490 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚166737281400832 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚168969284459232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3113742601894 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚174313931136512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _step‚359 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3123599453696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30957888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚172083113209712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚4449 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78831331 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192727040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚163522796590080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3214509714432 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45054233 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33777100 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45054233 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3214677537792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33777098 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚172083010021766 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3123599453696 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30957888 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚172083113209712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚4449 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚78831331 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 192727040 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚163522796590080 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3214509714432 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚45054233 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33777100 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45054233 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3214677537792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33777098 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚172083010021766 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30957648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚168969267419872 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚78841490 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33777100 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 167823360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚174314036205056 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33777098 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚78841490 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚166737281400832 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚168969284459232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3113742601894 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚174313931136512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _step‚359 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19777584 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176178142763008 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79676914 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3146982274332 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚1837.367196092 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚173021371706368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79687059 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165271859295232 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3248782241280 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚362 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34135827 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3156859078656 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19777587 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚173922894865970 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3156771056640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3156771056640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3156859078656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚173021388745728 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚176178247824384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SD¤"T7 DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3146896125858 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31288917 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45541087 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11511333 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196928512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚173021371706368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45551230 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚234 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165271834391552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚176178247824384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31289154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11511567 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚168520616632832 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0¨@ØüØ'óÒO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚170775998740112 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176178142763008 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79676914 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3146982274332 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚1837.367196092 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚173021371706368 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79687059 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165271859295232 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3248782241280 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚362 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34135827 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3156859078656 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19777587 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚173922894865970 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3156771056640 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3156771056640 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3156859078656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚173021388745728 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚237 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚176178247824384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3146896125858 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31288917 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45541087 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11511333 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196928512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚173021371706368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚45551230 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚234 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165271834391552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚176178247824384 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31289154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11511567 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚168520616632832 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19777584 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31295610 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19781874 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34143148 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 % _timestamp‚1.7845036838938534e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79704292 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3157449166336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34143150 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚173058755112960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31295855 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3249659679744 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3157537195520 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34143148 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚176216292308480 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚173960442885174 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚176216187239936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚173058755112960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚170812870807872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _runtime‚1837.957210763 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45561142 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79704292 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3147658225888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3147572077302 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚168557006725120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0€Ïž¾&öÝM PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚168557006725120 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3157449166336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19781877 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚170812887847232 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚79694133 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚173058738073600 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3157537195520 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34143150 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31295610 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19781874 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34143148 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 % _timestamp‚1.7845036838938534e+09 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚79704292 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3157449166336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34143150 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚173058755112960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31295855 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3249659679744 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3157537195520 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34143148 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚176216292308480 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚173960442885174 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚176216187239936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚173058755112960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚170812870807872 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _runtime‚1837.957210763 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚45561142 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79704292 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3147658225888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3147572077302 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_ñÑu%batch_end/segment.all.peak‚482ëî©€%2÷- _wandbruntime‚1837  _runtime‚1837 …Šøÿ /loss/FBt1/detection_loss‚1.9733838796615601 &loss/total_loss‚12.015908813836466 trainer/global_step‚4499 _step‚364 2loss/SRqqt2/assignment_loss‚0.7348501542839349 /loss/FRt1/detection_loss‚0.6725433068438201 /loss/FRt2/detection_loss‚1.4256762104759098 epoch‚5 % _timestamp‚1.7845036838949587e+09 _runtime‚1837.961598091 0loss/FRt2/assignment_loss‚1.0086891343008797 1loss/SRqqt1/assignment_loss‚0.765005418470706 1loss/FBt1/assignment_loss‚0.16272537505234747 1loss/SRqqt2/detection_loss‚0.7130730075061918 0loss/FBt2/assignment_loss‚0.0800011431727918 /loss/FBt2/detection_loss‚0.7988142486779936 /loss/FRt1/assignment_loss‚2.566123822870189 1loss/SRqqt1/detection_loss‚1.1150231125201424z½8 ù„ 0loss/FBt2/assignment_loss‚0.0800011431727918 /loss/FBt2/detection_loss‚0.7988142486779936 /loss/FRt1/assignment_loss‚2.566123822870189 1loss/SRqqt1/detection_loss‚1.1150231125201424 /loss/FBt1/detection_loss‚1.9733838796615601 &loss/total_loss‚12.015908813836466 trainer/global_step‚4499 _step‚364 2loss/SRqqt2/assignment_loss‚0.7348501542839349 /loss/FRt1/detection_loss‚0.6725433068438201 /loss/FRt2/detection_loss‚1.4256762104759098 epoch‚5 % _timestamp‚1.7845036838949587e+09 _runtime‚1837.961598091 0loss/FRt2/assignment_loss‚1.0086891343008797 1loss/SRqqt1/assignment_loss‚0.765005418470706 1loss/FBt1/assignment_loss‚0.16272537505234747 1loss/SRqqt2/detection_loss‚0.7130730075061918ìíáîqnú‚(:ã Š»õÒŒüô¿( disk./.usagePercent‚76.1323311064496 disk.dm-1.in‚0" disk.dm-1.out‚1207.57177734375 disk.nvme0n1.in‚ 87.6328125' disk.nvme1c1n1.out‚2791.01904296875 proc.cpu.threads‚17 disk.rbd0.out‚0 network.recv‚551326 disk.nvme1n1.in‚ 10.48828125 disk.md0.out‚48.84375 disk.dm-0.in‚ 98.12109375' disk.nvme0c0n1.out‚13245.1533203125# proc.memory.rssMB‚ 2873.30078125* proc.memory.percent‚17.537236213684082% disk./.usageGB‚1168.8216133117676 disk.md0.in‚0 disk.nvme0c0n1.in‚ 87.6328125% disk.nvme0n1.out‚13245.1533203125! disk.nvme1c1n1.in‚ 10.48828125$ memory_percent‚98.80926609039307 cpu‚26.59812932599239 network.sent‚ 5.854649e+06% disk.nvme1n1.out‚2791.01904296875" disk.dm-0.out‚14723.7763671875 disk.rbd0.in‚0( proc.memory.availableMB‚ 195.08984375ößéNû‚(:„ Š»õÒ¯¡À gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚111.258' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.086 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä™n2ü- _wandbruntime‚1857  _runtime‚1857‹u\ Éýà ' lr-AdamW/pg1‚0.00010962302773438515 ' lr-AdamW/pg2‚0.00010962302773438515 trainer/global_step‚4549 % _timestamp‚1.7845037039138496e+09 _runtime‚1857.977619192 _step‚365ìuüdÎþÈ % _timestamp‚1.7845037039138496e+09 _runtime‚1857.977619192 _step‚365 ' lr-AdamW/pg1‚0.00010962302773438515 ' lr-AdamW/pg2‚0.00010962302773438515 trainer/global_step‚4549í°f$2ÿ- _wandbruntime‚1857  _runtime‚1857âHDÚÔ'€ÎO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚170340525228032 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3283727656960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚1857.977678752 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3180725601266 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚172619602128112 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚178080503927808 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19992087 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34501877 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46037839 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11635020 % _timestamp‚1.7845037039145477e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚178080398866432 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3190796820480 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3190708798464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚172619619167472 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46047982 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80549861 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚175800327729378 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46037839 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚170340319910912 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚4549 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚178080398866432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3180811749740 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31626858 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31627107 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80539716 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174889707107328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174889707107328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3190708798464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11634774 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚167056617157632 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34501879 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 _step‚366 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚178080398866432 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3190796820480 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3190708798464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚172619619167472 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚46047982 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚80549861 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚175800327729378 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46037839 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚170340319910912 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚4549 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚178080398866432 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3180811749740 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31626858 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31627107 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚80539716 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚174889707107328 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚174889707107328 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3190708798464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11634774 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚167056617157632 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34501879 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 _step‚366 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚1857.977678752 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3180725601266 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚172619602128112 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚178080503927808 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19992087 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34501877 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46037839 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11635020 % _timestamp‚1.7845037039145477e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208î¯ ö2‚- _wandbruntime‚1858  _runtime‚1858pÍBa"ƒØM K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80567094 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚174927073474560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46047737 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31633658 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80556935 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46057894 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚174927056435200 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚167092287411200 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚175837875748582 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34509198 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3181487701296 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19996374 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3284428035584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3181401552710 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46047737 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80567094 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _runtime‚1858.248311426 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31633908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19996377 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11637284 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚170376922853888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34509200 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3191386908160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3191386908160 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46057894 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34509200 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 _step‚367 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34509198 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3191474937344 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚170376715446784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚178118443343360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 trainer/global_step‚4549 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ;4DeviceStatsMonitor.on_train_batch_end/8¯8d}max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 % _timestamp‚1.7845037041849465e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚174927056435200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3191474937344 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 m)Ïã&„ÝM O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19996374 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3284428035584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3181401552710 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46047737 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚80567094 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _runtime‚1858.248311426 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31633908 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19996377 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11637284 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚170376922853888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34509200 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3191386908160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3191386908160 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46057894 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34509200 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 _step‚367 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34509198 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3191474937344 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚170376715446784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚178118443343360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 trainer/global_step‚4549 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 % _timestamp‚1.7845037041849465e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚174927056435200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3191474937344 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚80567094 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚174927073474560 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46047737 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31633658 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚80556935 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46057894 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚174927056435200 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚167092287411200 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚175837875748582 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34509198 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3181487701296ïTËjì2…- _wandbruntime‚1858  _runtime‚1858iŽ†ý epoch‚5 _step‚368 2loss/SRqqt1/assignment_loss‚0.6778834653737729 1loss/FBt2/assignment_loss‚0.03442048592026136 1loss/SRqqt2/detection_loss‚0.7084645675981119 /loss/FRt1/assignment_loss‚2.693528361226383 1loss/FBt1/assignment_loss‚0.10836183264534469 /loss/FRt2/detection_loss‚1.5299388335480972 1loss/SRqqt1/detection_loss‚1.0978039123742662 .loss/FBt1/detection_loss‚1.426939642429352 %loss/total_loss‚11.20199285289062 trainer/global_step‚4549 % _timestamp‚1.7845037041863928e+09 0loss/FRt2/assignment_loss‚1.0365860307489643 1loss/SRqqt2/assignment_loss‚0.718784003662148 .loss/FBt2/detection_loss‚0.558934405730143 _runtime‚1858.249356897 /loss/FRt1/detection_loss‚0.6103473116337753m‡‚ .loss/FBt1/detection_loss‚1.426939642429352 %loss/total_loss‚11.20199285289062 trainer/global_step‚4549 % _timestamp‚1.7845037041863928e+09 0loss/FRt2/assignment_loss‚1.0365860307489643 1loss/SRqqt2/assignment_loss‚0.718784003662148 .loss/FBt2/detection_loss‚0.558934405730143 _runtime‚1858.249356897 /loss/FRt1/detection_loss‚0.6103473116337753 epoch‚5 _step‚368 2loss/SRqqt1/assignment_loss‚0.6778834653737729 1loss/FBt2/assignment_loss‚0.03442048592026136 1loss/SRqqt2/detection_loss‚0.7084645675981119 /loss/FRt1/assignment_loss‚2.693528361226383 1loss/FBt1/assignment_loss‚0.10836183264534469 /loss/FRt2/detection_loss‚1.5299388335480972 1loss/SRqqt1/detection_loss‚1.0978039123742662ðhoF‹jˆ‚(:ß ™»õÒ‰ð¿ disk.md0.in‚0 disk.dm-0.in‚ 98.12109375# disk.dm-0.out‚14732.34716796875 disk.rbd0.in‚0$ proc.memory.availableMB‚194.8125 cpu‚26.603081244914122 proc.cpu.threads‚17 network.recv‚555634& disk.nvme0n1.out‚13252.19775390625& disk.nvme1c1n1.out‚2792.9755859375$ memory_percent‚98.81095886230469) disk./.usagePercent‚76.13234725726036% disk./.usageGB‚1168.8218612670898 disk.nvme0n1.in‚ 87.6328125! disk.nvme1c1n1.in‚ 10.48828125 disk.nvme0c0n1.in‚ 87.6328125# proc.memory.rssMB‚ 2873.30078125* proc.memory.percent‚17.537236213684082 network.sent‚ 5.9045e+06 disk.nvme1n1.in‚ 10.48828125$ disk.nvme1n1.out‚2792.9755859375 disk.md0.out‚ 48.98828125 disk.dm-1.in‚0 disk.dm-1.out‚ 1207.6640625 disk.rbd0.out‚0( disk.nvme0c0n1.out‚13252.19775390625›çg‰‚(:… ™»õÒæϲÀ gpu.0.gpu‚47.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚120.621' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.207 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01˜!ØnŠ‚(:ã ¨»õÒåÇÝÀ) disk./.usagePercent‚76.15198018282598! disk.nvme1c1n1.in‚ 10.48828125$ disk.nvme1n1.out‚2793.6044921875# disk.dm-0.out‚14795.58935546875# proc.memory.rssMB‚ 2873.30078125& disk.nvme0n1.out‚13315.73291015625& disk.nvme1c1n1.out‚2793.6044921875 network.recv‚560303 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 87.6328125( disk.nvme0c0n1.out‚13315.73291015625 disk.dm-0.in‚ 98.12109375 disk.nvme1n1.in‚ 10.48828125 disk.md0.in‚0 disk.md0.out‚ 49.34765625 disk.dm-1.in‚0$ memory_percent‚98.80948066711426' proc.memory.availableMB‚ 195.0546875* proc.memory.percent‚17.537236213684082 cpu‚26.606737056646306 network.sent‚ 5.968801e+06$ disk./.usageGB‚1169.123275756836 disk.nvme0n1.in‚ 87.6328125 disk.dm-1.out‚ 1207.8046875 proc.cpu.threads‚172)0ì‹‚(:… ¨»õÒœž‹Â gpu.0.gpu‚53.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.768' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.256 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦ê'2Œ- _wandbruntime‚1878  _runtime‚1878¹VyÀÆÀ _runtime‚1878.684586641 _step‚369 & lr-AdamW/pg1‚0.0001095562921364778 & lr-AdamW/pg2‚0.0001095562921364778 trainer/global_step‚4599 $ _timestamp‚1.784503724620889e+09ƒœxËŽÅ & lr-AdamW/pg2‚0.0001095562921364778 trainer/global_step‚4599 $ _timestamp‚1.784503724620889e+09 _runtime‚1878.684586641 _step‚369 & lr-AdamW/pg1‚0.0001095562921364778ñÒ¾^2- _wandbruntime‚1878  _runtime‚1878s[‚Ô'ÎO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31963796 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 _step‚370 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34867929 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31964032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚168841350116352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3318892854784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3318718732800 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3224734562304 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3224646540288 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34867927 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚174463205516112 trainer/global_step‚4599 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚176758008429568 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚168841375020032 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚176758025468928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚176758008429568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81412663 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20206584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚172160267874816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚1878.684754251 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46534591 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46534591 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179982654969856 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚176758025468928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11757445 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚179982760031232 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46544734 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34867927 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3224646540288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208m ç.9‘ÓO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179982654969856 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚176758025468928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11757445 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚179982760031232 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚46544734 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batMèa»  ch_start/active.small_pool.peak‚12303 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚34867927 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3224646540288 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31963796 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 _step‚370 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34867929 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31964032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚168841350116352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3318892854784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3318718732800 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3224734562304 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3224646540288 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34867927 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚174463205516112 trainer/global_step‚4599 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚176758008429568 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚168841375020032 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚176758025468928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚176758008429568 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚81412663 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20206584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚172160267874816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚1878.684754251 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46534591 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46534591ò-µM¸2’- _wandbruntime‚1878  _runtime‚1878O†‡Þ&“ØM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34875248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚176795374796800 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚180020804515328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34875248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3215231028118 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚180020699446784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3225324649984 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34875250 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚168877070177280 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81419737 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚177715308611990 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20210877 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11759685 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3215317176704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3225412679168 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚172196669938688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚234 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11759916 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚177715411799936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46554646 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4599 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚180020804515328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81429896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚174500094623232 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31970793 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81429896 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46544489 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20210874 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3225412679168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168877045273600 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚176795391836160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46554646 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3319599761408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚174500077583872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚176795374796800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31970559 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3319417257984 _step‚371 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3215231028118 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚180020699446784 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3225324649984 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34875250 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚168877070177280 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81419737 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚177715308611990 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20210877 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11759685 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3215317176704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3225412679168 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚172196669938688 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚234 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11759916 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚177715411799936 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚46554646 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4599 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚180020804515328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81429896 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚174500094623232 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚31970793 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81429896 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚46544489 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20210874 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3225412679168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168877045273600 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚176795391836160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46554646 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3319599761408 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚174500077583872 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚176795374796800 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31970559 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3319417257984 _step‚371 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181884911073280 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35233979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11880562 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3258584282112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚179555296644028 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚178626326791168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3258672304128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20421087 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _step‚374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35233977 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35233979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35233977 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚181884911073280 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47031343 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32301649 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚173980004967936 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181885016134656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3248470700556 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚1899.461283544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82275465 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3258672304128µÄ÷ŠÙ'ŸÓO O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35233979 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35233977 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚181884911073280 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚47031343 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32301649 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚173980004967936 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚181885016134656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3248470700556 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚1899.461283544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚82275465 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3258672304128 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚181884911073280 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35233979 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11880562 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3258584282112 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚179555296644028 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚178626326791168 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3258672304128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20421087 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _step‚374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35233977 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145öb¥P2 - _wandbruntime‚1899  _runtime‚1899™0)HÞ&¡ØM O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35241298 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32308334 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚176343698011232 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82282539 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3259262391808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3259350420992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚178663710197760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚176343680971872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 % _timestamp‚1.7845037456720507e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82292698 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3354572293120 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3259350420992 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82292698 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35241298 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚179592741475398 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35241300 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32308091 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47051398 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚181923060618752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚178663693158400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181923060618752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3259262391808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47041241 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47051398 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3249146652112 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚178663693158400 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚174016400332800 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20425377 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚174016188731392 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170661803136000 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚179592844663344 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚1899.735434575 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35241300 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚178663710197760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82282539 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚170661828039680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3249060503526 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚375 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11882957 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181922955550208 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4649 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20425374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482·òEì¢ÝM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4649 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20425374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35241298 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32308334 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚176343698011232 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚82282539 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3259262391808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3259350420992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚178663710197760 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚176343680971872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 % _timestamp‚1.7845037456720507e+09 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚82292698 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3354572293120 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3259350420992 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚82292698 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35241298 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚179592741475398 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35241300 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32308091 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47051398 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚181923060618752 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚178663693158400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚181923060618752 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3259262391808 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47041241 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47051398 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3249146652112 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚178663693158400 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚174016400332800 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20425377 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚174016188731392 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚170661803136000 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚179592844663344 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_tr+éΊ÷ain_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚1899.735434575 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35241300 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚178663710197760 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚82282539 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚170661828039680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3249060503526 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚375 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11882957 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚181922955550208 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248÷Òˆ2£- _wandbruntime‚1899  _runtime‚1899b?ɤÿ /loss/FBt1/detection_loss‚1.4433562532067299 % _timestamp‚1.7845037456736329e+09 _step‚376 /loss/FRt1/assignment_loss‚2.726283657266416 trainer/global_step‚4649 0loss/FRt2/assignment_loss‚1.2416091534224423 1loss/SRqqt1/detection_loss‚1.3197446551712366 &loss/total_loss‚12.332506215108756 _runtime‚1899.736694587 2loss/SRqqt1/assignment_loss‚0.8273224123598387 1loss/FBt1/assignment_loss‚0.38234712478078514 1loss/FBt2/assignment_loss‚0.04352061127029856 .loss/FRt1/detection_loss‚0.586325186442456 .loss/FRt2/detection_loss‚1.494798711046946 /loss/FBt2/detection_loss‚0.8462008294718014 epoch‚5 2loss/SRqqt2/assignment_loss‚0.7145120272947273 1loss/SRqqt2/detection_loss‚0.7064855933750788‡t× ¥„ 2loss/SRqqt1/assignment_loss‚0.8273224123598387 1loss/FBt1/assignment_loss‚0.38234712478078514 1loss/FBt2/assignment_loss‚0.04352061127029856 .loss/FRt1/detection_loss‚0.586325186442456 .loss/FRt2/detection_loss‚1.494798711046946 /loss/FBt2/detection_loss‚0.8462008294718014 epoch‚5 2loss/SRqqt2/assignment_loss‚0.7145120272947273 1loss/SRqqt2/detection_loss‚0.7064855933750788 /loss/FBt1/detection_loss‚1.4433562532067299 % _timestamp‚1.7845037456736329e+09 _step‚376 /loss/FRt1/assignment_loss‚2.726283657266416 trainer/global_step‚4649 0loss/FRt2/assignment_loss‚1.2416091534224423 1loss/SRqqt1/detection_loss‚1.3197446551712366 &loss/total_loss‚12.332506215108756 _runtime‚1899.736694587ø1‘*Ìb¦‚(:× Æ»õÒ¹˜±À$ disk.nvme1c1n1.out‚ 2829.80078125" disk.nvme1n1.out‚ 2829.80078125# memory_percent‚98.7647294998169 cpu‚26.613286133197644 disk.nvme0n1.in‚ 87.6328125# disk.nvme0n1.out‚13502.35546875! disk.nvme1c1n1.in‚ 10.48828125 disk.dm-1.in‚0 disk.dm-1.out‚ 1208.05078125 disk.rbd0.out‚0 disk.md0.in‚0 disk.md0.out‚ 50.31640625) disk./.usagePercent‚76.14784482984818 disk.nvme1n1.in‚ 10.48828125 disk.dm-0.out‚15016.09765625( proc.memory.availableMB‚ 202.38671875* proc.memory.percent‚17.097902297973633 proc.cpu.threads‚17 network.recv‚567855% disk./.usageGB‚1169.0597877502441 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 87.6328125% disk.nvme0c0n1.out‚13502.35546875 disk.dm-0.in‚ 98.12109375" proc.memory.rssMB‚ 2801.3203125 network.sent‚ 6.034078e+06†¨y‘§‚(:† Æ»õÒ±ÌóÀ gpu.0.gpu‚89.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚200.739' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚66.913 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k¼c:f¨‚(:Û Õ»õÒ£°Û¿ disk.md0.out‚ 50.61328125 disk.dm-0.in‚ 98.12109375 disk.dm-0.out‚15023.28515625 disk.nvme0c0n1.in‚ 87.6328125 proc.cpu.threads‚17 disk.nvme1n1.in‚ 10.48828125! disk.dm-1.out‚1208.1474609375% disk.nvme0c0n1.out‚13508.12109375$ memory_percent‚98.76484870910645) disk./.usagePercent‚76.14784582374422% disk./.usageGB‚1169.0598030090332& disk.nvme1c1n1.out‚2831.9677734375 disk.nvme0n1.in‚ 87.6328125# disk.nvme0n1.out‚13508.12109375" proc.memory.rssMB‚ 2801.3203125 cpu‚26.61384833062528 network.sent‚ 6.09845e+06 network.recv‚571967$ disk.nvme1n1.out‚2831.9677734375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 202.3671875* proc.memory.percent‚17.097902297973633! disk.nvme1c1n1.in‚ 10.48828125 disk.md0.in‚0}ÙÆG›©‚(: Õ»õÒäâŒÀ gpu.0.gpu‚53.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚198.776' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.25866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0♥2ª- _wandbruntime‚1920  _runtime‚1920¾èúÈ«Â ' lr-AdamW/pg2‚0.00010942025163675563 trainer/global_step‚4699 $ _timestamp‚1.784503765967925e+09 _runtime‚1920.031085418 _step‚377 ' lr-AdamW/pg1‚0.00010942025163675563QZ`–Í¬Ç ' lr-AdamW/pg1‚0.00010942025163675563 ' lr-AdamW/pg2‚0.00010942025163675563 trainer/global_step‚4699 $ _timestamp‚1.784503765967925e+09 _runtime‚1920.031085418 _step‚377ùXÏ2­- _wandbruntime‚1920  _runtime‚1920SÁÞñÔ'®ÎO BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183787167176704 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚183787272238080 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚180494662192128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3292522023936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47528095 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _step‚378 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚181432626319602 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12003497 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚178150429331472 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20635584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3282300175964 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚180494645152768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚180494662192128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3388851602944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚181432729507436 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47538238 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35600029 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _runtime‚1920.033566582 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47538238 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32639084 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35600027 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3292522023936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35600027 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83138267 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 trainer/global_step‚4699 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚183787272238080 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚180494645152768 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172410890744832 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚172410865841152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20635587µî¼IÙ'¯ÓO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47528095 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _step‚378 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚181432626319602 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12003497 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚178150429331472 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20635584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3282300175964 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚180494645152768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚180494662192128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3388851602944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚181432729507436 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47538238 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35600029 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _runtime‚1920.033566582 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚47538238 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32639084 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35600027 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3292522023936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35600027 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83138267 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 trainer/global_step‚4699 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚183787272238080 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚180494645152768 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172410890744832 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚172410865841152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20635587 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183787167176704 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚183787272238080 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚180494662192128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3292522023936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0úæp<â2°- _wandbruntime‚1920  _runtime‚1920£Ó’ÅŒ±ØM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32645760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚183825211653632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚183825316722176 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47548150 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32645520 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3293200133632 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚180532011520000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚175835935515648 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35607348 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47548150 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20639874 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3389374517248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚1920.626702197 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚175836138728448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚178187301399232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 _step‚379 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚180532028559360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 % _timestamp‚1.7845037665633717e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚180532011520000 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3293200133632 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20639877 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83155500 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12005646 trainer/global_step‚4699 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83155500 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3293288162816 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚181470174338806 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3389552826368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35607350 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47537993 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83145341 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12005883 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482e¡ïã&²ÝM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚183825211653632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚183825316722176 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚47548150 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32645520 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3293200133632 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚180532011520000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚175835935515648 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35607348 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47548150 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20639874 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3389374517248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚1920.626702197 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚175836138728448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚178187301399232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 _step‚379 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚180532028559360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 % _timestamp‚1.7845037665633717e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚180532011520000 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3293200133632 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20639877 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83155500 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12005646 trainer/global_step‚4699 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83155500 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3293288162816 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚181470174338806 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3389552826368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35607350 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚47537993 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83145341 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12005883 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32645760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0û’¥¨2³- _wandbruntime‚1920  _runtime‚1920ût¼Ê´ÿ /loss/FRt1/detection_loss‚0.5926911889568042 $ _timestamp‚1.784503766564492e+09 _step‚380 /loss/FBt1/detection_loss‚1.1168185902376233 epoch‚5 0loss/FRt1/assignment_loss‚2.6589056985997472 0loss/FBt2/assignment_loss‚0.0529063380456715 1loss/SRqqt1/detection_loss‚1.2283375176574878 _runtime‚1920.627359039 2loss/SRqqt1/assignment_loss‚0.6517325592404363 /loss/FBt2/detection_loss‚0.8368396142052692 &loss/total_loss‚11.694494846455523 0loss/FRt2/assignment_loss‚1.1917855222568368 2loss/SRqqt2/assignment_loss‚0.6557659238986582 0loss/FBt1/assignment_loss‚0.4388370249489147 /loss/FRt2/detection_loss‚1.4990007865262944 1loss/SRqqt2/detection_loss‚0.7708740818817802 trainer/global_step‚4699g<=½ µ„ 2loss/SRqqt1/assignment_loss‚0.6517325592404363 /loss/FBt2/detection_loss‚0.8368396142052692 &loss/total_loss‚11.694494846455523 0loss/FRt2/assignment_loss‚1.1917855222568368 2loss/SRqqt2/assignment_loss‚0.6557659238986582 0loss/FBt1/assignment_loss‚0.4388370249489147 /loss/FRt2/detection_loss‚1.4990007865262944 1loss/SRqqt2/detection_loss‚0.7708740818817802 trainer/global_step‚4699 /loss/FRt1/detection_loss‚0.5926911889568042 $ _timestamp‚1.784503766564492e+09 _step‚380 /loss/FBt1/detection_loss‚1.1168185902376233 epoch‚5 0loss/FRt1/assignment_loss‚2.6589056985997472 0loss/FBt2/assignment_loss‚0.0529063380456715 1loss/SRqqt1/detection_loss‚1.2283375176574878 _runtime‚1920.627359039ü!}GMp¶‚(:å ä»õÒÿ§“À# proc.memory.rssMB‚ 2801.63671875 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚13517.50537109375 proc.cpu.threads‚17 network.sent‚ 6.114257e+06) disk./.usagePercent‚76.14783812104986 disk.nvme0n1.in‚ 87.6328125! disk.nvme1c1n1.in‚ 10.48828125& disk.nvme1c1n1.out‚2832.7060546875$ disk.nvme1n1.out‚2832.7060546875( proc.memory.availableMB‚ 201.76953125& disk.nvme0n1.out‚13517.50537109375 disk.dm-0.in‚ 98.12109375! disk.dm-1.out‚1208.3193359375# memory_percent‚98.7684965133667* proc.memory.percent‚17.099833488464355 cpu‚26.614114018226864 network.recv‚575400$ disk./.usageGB‚1169.059684753418 disk.nvme1n1.in‚ 10.48828125 disk.md0.out‚ 51.0703125# disk.dm-0.out‚15032.25927734375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 87.6328125bÙ›·‚(: ä»õÒ¸¾À gpu.0.gpu‚87.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚197.81' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.93666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸¿¼ã2¸- _wandbruntime‚1940  _runtime‚1940¿‡XǹÁ trainer/global_step‚4749 % _timestamp‚1.7845037868048685e+09 _runtime‚1940.868548233 _step‚381 & lr-AdamW/pg1‚0.0001093509489139055 & lr-AdamW/pg2‚0.0001093509489139055"´ýìÌºÆ & lr-AdamW/pg1‚0.0001093509489139055 & lr-AdamW/pg2‚0.0001093509489139055 trainer/global_step‚4749 % _timestamp‚1.7845037868048685e+09 _runtime‚1940.868548233 _step‚381ýÌÈ%©2»- _wandbruntime‚1940  _runtime‚1940U>×üÔ'¼ÎO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3326547787776 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚1940.868718663 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚182362980553728 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20850087 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32976086 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚179994032719472 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35966077 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185689423280128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3423799784960 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚182362963514368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3326459765760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20850084 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174195648607232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3326547787776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12126239 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚177619247269376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35966079 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185689528341504 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48024847 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3326459765760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3316129651372 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35966079 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚183310162370844 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12126002 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 % _timestamp‚1.7845037868056068e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201122816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚185689423280128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3423623565824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48024847 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185689528341504 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35966077 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32976326 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚240 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚182362963514368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚183310059183010 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚4749 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48034990 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84001069 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚382 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83990924 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚182362980553728 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48034990 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚179994015680112 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚185689423280128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3423623565824 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48024847 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185689528341504 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35966077 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32976326 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚240 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚182362963514368 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚183310059183010 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚4749 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48034990 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84001069 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚382 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚83990924 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚182362980553728 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.alloc[&¾€ated‚48034990 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚179994015680112 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185689423280128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3423799784960 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚182362963514368 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3326459765760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20850084 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174195648607232 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3326547787776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12126239 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚177619247269376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚35966079 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185689528341504 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48024847 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3326459765760 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3316129651372 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35966079þ€Zîd2¾- _wandbruntime‚1941  _runtime‚1941*(Ý&¿×M J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35973398 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚183347710390160 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84018302 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32982712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3316805602928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚182400329881600 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20854377 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3424500013056 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12128575 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84018302 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35973400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84008143 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48044902 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48044902 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174231343764480 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _runtime‚1941.142441873 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3327225904640 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185727572825600 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20854374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177655632176128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _step‚383 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84008143 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185727467757056 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174231318860800 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚182400346920960 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3327137875456 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35973398 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808µ\Çâ&ÀÜM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚180030887747872 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185727572825600 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32982712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3316805602928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚182400329881600 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20854377 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3424500013056 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12128575 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84018302 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚35973400 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84008143 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48044902 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48044902 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚174231343764480 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _runtime‚1941.142441873 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3327225904640 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185727572825600 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20854374 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177655632176128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _step‚383 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84008143 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185727467757056 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174231318860800 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚182400346920960 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3327137875456 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35973398 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35973398 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚183347710390160 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84018302 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚185187595234252 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48531742 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184231281875968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21064584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚175980381565952 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3458591500800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4799 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚187591679383552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactÏîive_split_bytes.large_pool.peak‚ 743477248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3349959126780 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚184231298915328 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84863871 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187591679383552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3349872978306 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3360485529600 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3360397507584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36332127 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚187591784444928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚181837619068112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12249351 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚184231298915328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21064587 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184231281875968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚179438973066752 % _timestamp‚1.7845038074328325e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚179439182578176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3458776108544 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3360397507584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48521599 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33313697 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚175980406469632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚181837636107472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48521599 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680B`5™Ù'ËÓO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚84853726 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33313938 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36332129 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚48531742 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚187591784444928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚185187492046418 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 _runtime‚1961.496013284 _step‚386 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚185187595234252 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48531742 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184231281875968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21064584 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚175980381565952 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3458591500800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4799 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚187591679383552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3349959126780 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚184231298915328 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚84863871 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚187591679383552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3349872978306 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3360485529600 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3360397507584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚36332127 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚187591784444928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚181837619068112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12249351 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚184231298915328 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21064587 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184231281875968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚179438973066752 % _timestamp‚1.7845038074328325e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚179439182578176 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3458776108544 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3360397507584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48521599 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33313697 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚175980406469632 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚181837636107472 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48521599 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 IDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33320368 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3361163646464 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚187629828929024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3459294144512 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84881104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚176016076723200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36339450 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179475370867712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179475576177664 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 _step‚387 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3361163646464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚184268648243200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚185225143253568 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84881104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3361075617280 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48541654 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21068874 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3350635078336 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84870945 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48541654 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚184268665282560 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36339448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3459474550784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12251739 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48531497 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚184268665282560 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84870945 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚185225040065622 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚181874508175232 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚187629828929024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0àNe ÎÚM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36339448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 _runtime‚ 1961.76888687 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚48541654 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚184268665282560 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36339448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3459474550784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12251739 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48531497 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚184268665282560 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚84870945 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚185225040065622 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚181874508175232 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚187629828929024 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 # _timestamp‚1.78450380770556e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33320368 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3361163646464 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitorjóQµ× .on_train_batch_end/segment.large_pool.peak‚330 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚187629828929024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3459294144512 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚84881104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚176016076723200 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36339450 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179475370867712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179475576177664 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 _step‚387 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3361163646464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚184268648243200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚185225143253568 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚84881104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3361075617280 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚48541654 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21068874 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3350635078336 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚84870945 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 Fב| =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9880732297897339 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4830729067325592 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6000000238418579 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9796064496040344 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9917590022087097 _runtime‚1981.435657466 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.896813690662384 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12785811722278595 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07787700742483139 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.017623620107769966 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.968563973903656 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.575323224067688 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.08439980447292328 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3467576801776886 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.030760055407881737 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.12024751305580139 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20931881666183472 ,jet/accuracy_3_of_3‚0.054993584752082825 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45347949862480164 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.12493091821670532 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7090949416160583 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9898989200592041 'particle/f_score‚0.6278311014175415 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9971655011177063 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5887360572814941 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11226765066385269 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983673095703125 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.13937465846538544 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19350022077560425 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1157292053103447 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6800000071525574 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9529427886009216 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0778929591178894 jet/accuracy_3_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9932278394699097 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9764714241027832 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7661027312278748 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3863636255264282 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7502860426902771 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.752767026424408 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.862500011920929 trainer/global_step‚4805 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4639226496219635 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03804143890738487 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.435791552066803 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6808419823646545 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3214285671710968 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1152590811252594 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.12837783992290497 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3395061492919922 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975759387016296 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6447659730911255 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03207113593816757 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 epoch‚5 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5625748038291931 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.12827511131763458 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6800000071525574 *jet/accuracy_1_of_1‚0.2711540758609772 ,jet/accuracy_4_of_4‚0.014682539738714695 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988389015197754 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.686073362827301 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.20000000298023224 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3194444477558136 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5793631672859192 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 *jet/accuracy_2_of_4‚0.7706965208053589 particle/accuracy_2_of_3‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5819355845451355 particle/accuracy_1_of_1‚1 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11154013872146606 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5380368232727051 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09095671772956848 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6457126140594482 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.990031898021698 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1157292053103447 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13124999403953552 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_2‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9953703880310059 *jet/accuracy_1_of_4‚0.9655982255935669 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7959432005882263 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08038247376680374 /particle/accuracy_4_of_4‚0.9478905200958252 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.11621948331594467 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9862064123153687 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9917260408401489 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21172277629375458 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6402788758277893 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5390307307243347 +jet/accuracy_2_of_3‚0.48861390352249146 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7381258010864258 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3214285671710968 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.528184175491333 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7959432005882263 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5024569630622864 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983673095703125 % _timestamp‚1.7845038273720155e+09 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5226852297782898 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6993613243103027 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.12937480211257935 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9900141954421997 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5164923667907715 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009354551322758198 *jet/accuracy_1_of_3‚0.9064160585403442 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10124470293521881 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08038247376680374 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983881115913391 jet/accuracy_4_of_5‚0 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1508394479751587 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.44105857610702515 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.32898151874542236 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12196312844753265 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08240540325641632 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7093154191970825 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.20000000298023224 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9871794581413269 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8802679777145386 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9954544901847839 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.016943491995334625 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.637231171131134 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2504376769065857 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08231771737337112 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5859788060188293 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10697170346975327 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.49273568391799927 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8796961903572083 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9974656701087952 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6337607502937317 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7508772015571594 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13491375744342804 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.570118248462677 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.49157029390335083 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4650000035762787 +particle/sensitivity‚0.6174988746643066 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7validation_average_jet_accuracy‚0.32172536849975586 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6375353336334229 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3413800299167633 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9714908599853516 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3457585871219635 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.19430789351463318 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7207930684089661 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.48980727791786194 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3886255919933319 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 .particle/accuracy_3_of_3‚0.999514102935791 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8392857313156128 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5243055820465088 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11933554708957672 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7246626615524292 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_5‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_4‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9901315569877625 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.47330501675605774 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9044300317764282 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.18959659337997437 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13904030621051788 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.14809486269950867 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5887360572814941 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9807692170143127 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6449942588806152 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9963600635528564 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7178629636764526 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1552020162343979 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7350427508354187 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.33815523982048035 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7017823457717896 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_2_of_4‚1 particle/accuracy_2_of_5‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_4_of_5‚1 *loss/val_total_loss‚10.906929656297441 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8225268125534058 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.12651841342449188 (particle/accuracy‚0.8428160548210144 jet/accuracy_2_of_5‚0 _step‚389 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.613932728767395 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8313940167427063 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 *jet/accuracy_1_of_2‚0.7543033957481384 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9970238208770752 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5796016454696655 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 /particle/accuracy_2_of_2‚0.9998950958251953 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.680701732635498É7ý´D#Ø–| 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.48980727791786194 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.3886255919933319 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 .particle/accuracy_3_of_3‚0.999514102935791 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.8392857313156128 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5243055820465088 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.11933554708957672 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7246626615524292 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 particle/accuracy_3_of_5‚1 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_1_of_4‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9901315569877625 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.47330501675605774 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9044300317764282 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.18959659337997437 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.13904030621051788 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.14809486269950867 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5887360572814941 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9807692170143127 & Purity/2FRt1SRqqt2FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6449942588806152 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9963600635528564 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7178629636764526 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1552020162343979 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.7350427508354187 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.33815523982048035 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7017823457717896 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 particle/accuracy_2_of_4‚1 particle/accuracy_2_of_5‚1 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 particle/accuracy_4_of_5‚1 *loss/val_total_loss‚10.906929656297441 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8225268125534058 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_2‚0.12651841342449188 (particle/accuracy‚0.8428160548210144 jet/accuracy_2_of_5‚0 _step‚389 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.613932728767395 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8313940167427063 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 *jet/accuracy_1_of_2‚0.7543033957481384 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9970238208770752 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.5796016454696655 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 /particle/accuracy_2_of_2‚0.9998950958251953 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.680701732635498 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9880732297897339 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4830729067325592 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6000000238418579 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9796064496040344 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9917590022087097 _runtime‚1981.435657466 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.896813690662384 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.12785811722278595 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.07787700742483139 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.017623620107769966 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.968563973903656 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.575323224067688 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.08439980447292328 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3467576801776886 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.030760055407881737 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.12024751305580139 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20931881666183472 ,jet/accuracy_3_of_3‚0.054993584752082825 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45347949862480164 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.12493091821670532 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7090949416160583 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9898989200592041 'particle/f_score‚0.6278311014175415 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.625 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9971655011177063 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5887360572814941 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11226765066385269 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983673095703125 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.13937465846538544 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19350022077560425 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1157292053103447 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.6800000071525574 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9529427886009216 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0778929591178894 jet/accuracy_3_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9932278394699097 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9764714241027832 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.7661027312278748 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.3863636255264282 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7502860426902771 8"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.752767026424408 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.862500011920929 trainer/global_step‚4805 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4639226496219635 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03804143890738487 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.435791552066803 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.6808419823646545 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3214285671710968 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1152590811252594 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.12837783992290497 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3395061492919922 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975759387016296 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6447659730911255 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03207113593816757 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 epoch‚5 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 jet/accuracy_1_of_5‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.5625748038291931 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.12827511131763458 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.6800000071525574 *jet/accuracy_1_of_1‚0.2711540—¬¿Ø0758609772 ,jet/accuracy_4_of_4‚0.014682539738714695 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988389015197754 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.686073362827301 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.20000000298023224 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3194444477558136 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5793631672859192 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 *jet/accuracy_2_of_4‚0.7706965208053589 particle/accuracy_2_of_3‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5819355845451355 particle/accuracy_1_of_1‚1 + Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9375 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11154013872146606 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5380368232727051 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09095671772956848 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6457126140594482 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.990031898021698 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1157292053103447 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13124999403953552 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 particle/accuracy_1_of_2‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9953703880310059 *jet/accuracy_1_of_4‚0.9655982255935669 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.7959432005882263 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08038247376680374 /particle/accuracy_4_of_4‚0.9478905200958252 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.11621948331594467 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9862064123153687 particle/accuracy_1_of_5‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9917260408401489 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.21172277629375458 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6402788758277893 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5390307307243347 +jet/accuracy_2_of_3‚0.48861390352249146 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7381258010864258 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3214285671710968 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.528184175491333 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.7959432005882263 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5024569630622864 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983673095703125 % _timestamp‚1.7845038273720155e+09 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5226852297782898 particle/accuracy_5_of_5‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.6993613243103027 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.12937480211257935 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9900141954421997 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5164923667907715 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.009354551322758198 *jet/accuracy_1_of_3‚0.9064160585403442 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10124470293521881 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.08038247376680374 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9983881115913391 jet/accuracy_4_of_5‚0 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.1508394479751587 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.44105857610702515 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.32898151874542236 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.12196312844753265 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.08240540325641632 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7093154191970825 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.20000000298023224 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9871794581413269 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.8802679777145386 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9954544901847839 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.016943491995334625 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 ( Purity/2FRt1SRqqt2FBt/FBt_purity‚0.5 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.637231171131134 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2504376769065857 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.08231771737337112 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.5859788060188293 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10697170346975327 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.49273568391799927 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8796961903572083 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9974656701087952 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.6337607502937317 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.7508772015571594 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13491375744342804 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.570118248462677 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.49157029390335083 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4650000035762787 +particle/sensitivity‚0.6174988746643066 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7validation_average_jet_accuracy‚0.32172536849975586 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6375353336334229 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3413800299167633 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9714908599853516 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3457585871219635 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.19430789351463318 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7207930684089661…\ ÑþvÙ‚(:ë  ¼õÒØø¢Á% disk./.usageGB‚1168.8894424438477 disk.nvme0c0n1.in‚ 87.6328125' disk.nvme1c1n1.out‚2900.65478515625 network.recv‚592111) disk./.usagePercent‚76.13674922285281 disk.md0.out‚ 52.92578125# disk.dm-0.out‚15069.49267578125" disk.dm-1.out‚1272.64599609375$ memory_percent‚98.80363941192627* proc.memory.percent‚17.490696907043457 proc.cpu.threads‚17 network.sent‚ 6.293725e+06 disk.rbd0.out‚0 disk.nvme0n1.in‚ 87.6328125& disk.nvme0n1.out‚13555.06396484375 disk.nvme1n1.in‚ 10.48828125 disk.md0.in‚0 disk.dm-0.in‚ 98.12109375# proc.memory.rssMB‚ 2865.67578125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚13555.06396484375! disk.nvme1c1n1.in‚ 10.48828125% disk.nvme1n1.out‚2900.65478515625 disk.dm-1.in‚0( proc.memory.availableMB‚ 196.01171875 cpu‚26.656467740663945Ö³‚DÚ‚(:…  ¼õÒþűÁ gpu.0.gpu‚19.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚195.498' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚65.166 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨XÐ2Û- _wandbruntime‚1999  _runtime‚1999Ïif0ÉÜà _step‚390 ' lr-AdamW/pg1‚0.00010920978407232832 ' lr-AdamW/pg2‚0.00010920978407232832 trainer/global_step‚4849 % _timestamp‚1.7845038452160158e+09 _runtime‚1999.279215882Ç:k˜ÎÝÈ ' lr-AdamW/pg1‚0.00010920978407232832 ' lr-AdamW/pg2‚0.00010920978407232832 trainer/global_step‚4849 % _timestamp‚1.7845038452160158e+09 _runtime‚1999.279215882 _step‚390†’1ó2Þ- _wandbruntime‚1999  _runtime‚1999çcÅðÔ'ßÎO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3414755566592 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚178690562692096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3514260624896 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚190487019399168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3414843588608 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12451849 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 _runtime‚1999.280088693 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚188053911573578 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3514073920000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49185617 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33825013 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49195760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21373164 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3404182609754 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚187072263832576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚182204848220672 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49185617 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚190487124460544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21373167 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36935645 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86121262 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36935647 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚184649728963824 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3514073920000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49185617 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33825013 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49195760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21373164 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3404182609754 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚187072263832576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚182204848220672 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49185617 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚190487124460544 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21373167 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36935645 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚270 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86121262 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36935647 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚184649728963824 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚190487019399168 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3414843588608 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12451849 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 _runtime‚1999.280088693‡±ÿÏÆ2á- _wandbruntime‚1999  _runtime‚1999ƒ~û âØM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36942968 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49195515 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚190525168944640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3415521705472 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49205672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3514778361856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚184686583992224 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 _step‚392 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36942968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚190525063876096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚184686601031584 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86138481 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188091356404948 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49195515 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21377457 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚182241036211200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187109647239168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448¤‚ëiã NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36942966 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚187109647239168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚190525168944640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 trainer/global_step‚4849 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3415433676288 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36942966 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚187109630199808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3415433676288 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚178726257849344 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3404772412724 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49205672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86148640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12454397 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86148640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33831851 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86138481 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21377454 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚178726282753024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1999.550561328?þÙNã&ãÝM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12454397 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚272 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚86148640 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33831851 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86138481 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21377454 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚178726282753024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1999.550561328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36942968 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49195515 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚190525168944640 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3415521705472 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49205672 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3514778361856 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚184686583992224 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 _step‚392 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36942968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚190525063876096 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚184686601031584 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86138481 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188091356404948 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49195515 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21377457 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚182241036211200 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187109647239168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36942966 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚187109647239168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚190525168944640 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 trainer/global_step‚4849 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3415433676288 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36942966 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚187109630199808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3415433676288 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚275 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚178726257849344 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3404772412724 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49205672 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86148640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928ˆ-fd2ä- _wandbruntime‚1999  _runtime‚1999{Èå€ 0loss/FRt1/assignment_loss‚2.5304502235890465 2loss/SRqqt1/assignment_loss‚0.6893270387265243 1loss/FBt1/assignment_loss‚0.33999911837641916 &loss/total_loss‚10.894114235683816 _runtime‚1999.551231139 1loss/FBt2/assignment_loss‚0.05127287954025593 /loss/FRt2/detection_loss‚1.5033544157042928 /loss/FBt2/detection_loss‚0.6415591735906312 trainer/global_step‚4849 2loss/SRqqt2/assignment_loss‚0.6858225696961637 0loss/SRqqt2/detection_loss‚0.672393459623751 /loss/FBt1/detection_loss‚0.9535204961609901 epoch‚6 % _timestamp‚1.7845038454883053e+09 _step‚393 /loss/FRt2/assignment_loss‚1.026754858393322 /loss/FRt1/detection_loss‚0.6089836966800659 1loss/SRqqt1/detection_loss‚1.1906763056023537OæO æ… &loss/total_loss‚10.894114235683816 _runtime‚1999.551231139 1loss/FBt2/assignment_loss‚0.05127287954025593 /loss/FRt2/detection_loss‚1.5033544157042928 /loss/FBt2/detection_loss‚0.6415591735906312 trainer/global_step‚4849 2loss/SRqqt2/assignment_loss‚0.6858225696961637 0loss/SRqqt2/detection_loss‚0.672393459623751 /loss/FBt1/detection_loss‚0.9535204961609901 epoch‚6 % _timestamp‚1.7845038454883053e+09 _step‚393 /loss/FRt2/assignment_loss‚1.026754858393322 /loss/FRt1/detection_loss‚0.6089836966800659 1loss/SRqqt1/detection_loss‚1.1906763056023537 0loss/FRt1/assignment_loss‚2.5304502235890465 2loss/SRqqt1/assignment_loss‚0.6893270387265243 1loss/FBt1/assignment_loss‚0.33999911837641916‰ÅÈ>kç‚(:à ¯¼õÒžñˆÀ' disk.nvme0c0n1.out‚13582.8525390625 disk.nvme0n1.in‚ 87.6328125% disk.nvme1n1.out‚2965.09423828125' proc.memory.availableMB‚ 195.8203125# proc.memory.rssMB‚ 2865.73828125 cpu‚26.650894541793054 disk.rbd0.out‚0! disk.nvme1c1n1.in‚ 10.48828125 disk.nvme1n1.in‚ 10.48828125 disk.md0.out‚53.15625 network.sent‚ 6.363632e+06 network.recv‚596034% disk./.usageGB‚1168.9436531066895 disk.nvme0c0n1.in‚ 87.6328125% disk.nvme0n1.out‚13582.8525390625 disk.dm-0.in‚ 98.12109375 proc.cpu.threads‚17' disk./.usagePercent‚76.140280287034 disk.dm-1.in‚0" disk.dm-1.out‚1336.65771484375' disk.nvme1c1n1.out‚2965.09423828125 disk.md0.in‚0! disk.dm-0.out‚15097.189453125$ memory_percent‚98.80480766296387) proc.memory.percent‚17.49107837677002 disk.rbd0.in‚0„Ù"!œè‚(:‘ ¯¼õÒ¡Ù°À gpu.0.gpu‚89.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚203.314' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.77133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MþéO2é- _wandbruntime‚2019  _runtime‚2019-¼44Éêà ' lr-AdamW/pg2‚0.00010913792421464273 trainer/global_step‚4899 % _timestamp‚1.7845038653601801e+09 _runtime‚2019.423342966 _step‚394 ' lr-AdamW/pg1‚0.00010913792421464273£¹gÎëÈ ' lr-AdamW/pg2‚0.00010913792421464273 trainer/global_step‚4899 % _timestamp‚1.7845038653601801e+09 _runtime‚2019.423342966 _step‚394 ' lr-AdamW/pg1‚0.00010913792421464273ŠÑgB2ì- _wandbruntime‚2019  _runtime‚2019]ò\Ô'íÎO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49692512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34163127 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚192389275502592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3448781330432 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚184024393059840 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚4899 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚189931241249152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚186493332351824 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚192389275502592 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚189931344436986 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚186493315312464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3549255015936 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3448693308416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚192389380563968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12575213 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚188940599233536 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3448693308416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚192389380563968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3448781330432 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3438012085162 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49692512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚180475320554496 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 % _timestamp‚1.7845038653611789e+09 _runtime‚2019.427629193 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34162877 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34163127 ZKDeviceStatsMoniŽ‡tor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚192389275502592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3448781330432 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚184024393059840 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚4899 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚189931241249152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚186493332351824 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚192389275502592 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚189931344436986 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚186493315312464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3549255015936 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3448693308416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚192389380563968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12575213 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚188940599233536 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712‹¥ÛÚ2ï- _wandbruntime‚2019  _runtime‚2019˜&ˆÜ&ðÖM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87001283 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 $ _timestamp‚1.784503865633367e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49692267 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180511040615424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 _step‚396 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3449459447296 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87011442 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3449459447296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚188977965600768 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87011442 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37309016 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4899 _runtime‚ 2019.69666656 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12577923 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚192427319979520 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21591957 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37309018 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚192427425048064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚192427425048064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87001283 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3438601888132 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49692267 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚188977948561408 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚188977948561408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3449371418112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37309016 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37309018 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12577682 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34169636 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3438688036718 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚186530187380224 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49702424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49702424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3549772601344 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Sbá&ñÛM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37309018 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚192427425048064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚192427425048064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87001283 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3438601888132 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49692267 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚188977948561408 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚188977948561408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3449371418112 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37309016 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37309018 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12577682 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34169636 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3438688036718 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚186530187380224 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49702424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49702424 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3549772601344 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87001283 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 $ _timestamp‚1.784503865633367e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49692267 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚180511040615424 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 _step‚396 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3449459447296 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87011442 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3449459447296 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚188977965600768 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87011442 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37309016 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4899 _runtime‚ 2019.69666656 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12577923 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚192427319979520 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21591957 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712ŒZ8(<2ò- _wandbruntime‚2019  _runtime‚2019ÓŠ,ó _runtime‚2019.700584135 0loss/FRt1/assignment_loss‚2.5020831225995432 1loss/SRqqt2/detection_loss‚0.7069842120274521 /loss/FBt2/detection_loss‚0.6101987169305276 %loss/total_loss‚11.15284534681799 _step‚397 1loss/FBt1/assignment_loss‚0.17381623220773604 trainer/global_step‚4899 0loss/FRt2/assignment_loss‚1.2344035101711535 2loss/SRqqt1/assignment_loss‚0.7496534231468104 /loss/FRt2/detection_loss‚1.3724173240360236 0loss/SRqqt1/detection_loss‚1.280237365863286 epoch‚6 2loss/SRqqt2/assignment_loss‚0.7015418286204667 2loss/FBt2/assignment_loss‚0.019563665851998892 /loss/FRt1/detection_loss‚0.6098922236559334 /loss/FBt1/detection_loss‚1.1920537217070608 % _timestamp‚1.7845038656343865e+09¡ªP ô† epoch‚6 2loss/SRqqt2/assignment_loss‚0.7015418286204667 2loss/FBt2/assignment_loss‚0.019563665851998892 /loss/FRt1/detection_loss‚0.6098922236559334 /loss/FBt1/detection_loss‚1.1920537217070608 % _timestamp‚1.7845038656343865e+09 _runtime‚2019.700584135 0loss/FRt1/assignment_loss‚2.5020831225995432 1loss/SRqqt2/detection_loss‚0.7069842120274521 /loss/FBt2/detection_loss‚0.6101987169305276 %loss/total_loss‚11.15284534681799 _step‚397 1loss/FBt1/assignment_loss‚0.17381623220773604 trainer/global_step‚4899 0loss/FRt2/assignment_loss‚1.2344035101711535 2loss/SRqqt1/assignment_loss‚0.7496534231468104 /loss/FRt2/detection_loss‚1.3724173240360236 0loss/SRqqt1/detection_loss‚1.280237365863286 °ßoõ‚(:ä ¾¼õÒΚŸÀ network.recv‚600293% disk./.usageGB‚1169.1919326782227& disk.nvme1c1n1.out‚2965.8955078125 disk.md0.in‚0 disk.md0.out‚ 53.66796875# disk.dm-0.out‚15154.06591796875) disk./.usagePercent‚76.15645221808973$ disk.nvme1n1.out‚2965.8955078125 disk.dm-1.in‚0 disk.nvme0n1.in‚ 87.6328125% disk.nvme0n1.out‚13640.2099609375" disk.dm-1.out‚1336.84521484375 network.sent‚ 6.425978e+06! disk.nvme1c1n1.in‚ 10.48828125 disk.rbd0.out‚0' disk.nvme0c0n1.out‚13640.2099609375# memory_percent‚98.8051176071167) proc.memory.percent‚17.49107837677002 cpu‚26.64548856481704 disk.nvme1n1.in‚ 10.48828125 disk.dm-0.in‚ 98.12109375 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 87.6328125( proc.memory.availableMB‚ 195.76953125# proc.memory.rssMB‚ 2865.73828125 proc.cpu.threads‚17%N;ö‚(:… ¾¼õÒó¤ÍÀ gpu.0.gpu‚53.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚194.436' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚64.812 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09Âq52÷- _wandbruntime‚2039  _runtime‚2039ª¦³ Éøà ' lr-AdamW/pg1‚0.00010906521426245629 ' lr-AdamW/pg2‚0.00010906521426245629 trainer/global_step‚4949 % _timestamp‚1.7845038855088437e+09 _runtime‚2039.572573925 _step‚398®üJÎùÈ trainer/global_step‚4949 % _timestamp‚1.7845038855088437e+09 _runtime‚2039.572573925 _step‚398 ' lr-AdamW/pg1‚0.00010906521426245629 ' lr-AdamW/pg2‚0.00010906521426245629Ž»6=K2ú- _wandbruntime‚2039  _runtime‚2039˜SÔô]ûÎO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34500662 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚194291531606016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚185844336887808 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚188336918700464 _runtime‚2039.572640295 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚190808900555776 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚190808900555776 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚50179121 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50179121 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.allÖyVw .allocated‚194291636667392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚87857011 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚249 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3471841560570 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37667747 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12698498 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚182260103320576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3584233567232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87846866 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚190808917595136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188336935739824 trainer/global_step‚4949 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12698744 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚191808777300394 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50189264 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚194291531606016 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3482719072256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21802164 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚190808917595136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182260078416896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3471755412096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3584053153792 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34500911 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50189264 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚194291636667392 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34500911 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50189264 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚194291636667392 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3471841560570 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37667747 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12698498 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚182260103320576 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3584233567232 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚87846866 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚190808917595136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188336935739824 trainer/global_step‚4949 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12698744 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚191808777300394 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50189264 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚194291531606016 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3482719072256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21802164 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚190808917595136 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182260078416896 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3471755412096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3584053153792 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 IÞ&þØM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3483397189120 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172017664 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34507393 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21806454 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12700939 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50199176 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚191846325319710 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚194329681151488 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3584759565824 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 trainer/global_step‚4949 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87864085 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21806457 % _timestamp‚1.7845038857816064e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3483397189120 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3472431363540 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚188373807807584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37675068 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87874244 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3483309159936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚188373790768224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182295798477824 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3472517512126 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87864085 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚190846266923008 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37675066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37675068 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚185880730061312 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 196921344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚185880533139968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3483309159936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3584931583488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12701178 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚194329576082944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87874244 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 _runtime‚2039.844885327 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚191846222131764 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50189019 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34507635 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚190846283962368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50189019 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _step‚400À«km}ÿÝM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34507393 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21806454 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12700939 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50199176 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚191846325319710 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚194329681151488 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3584759565824 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 trainer/global_step‚4949 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚87864085 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21806457 % _timestamp‚1.7845038857816064e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3483397189120 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3472431363540 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚188373807807584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚37675068 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚87874244 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3483309159936 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚188373790768224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚182295798477824 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3472517512126 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 KEãÌwÈfDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚87864085 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚190846266923008 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚37675066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚37675068 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚185880730061312 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 196921344 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚185880533139968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3483309159936 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3584931583488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12701178 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚194329576082944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚87874244 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 _runtime‚2039.844885327 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚191846222131764 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚50189019 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34507635 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚190846283962368 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50189019 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _step‚400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3483397189120 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172017664 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50686016 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88719813 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚193686210163802 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3505584887504 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚196193787709440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38033795 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34839291 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 trainer/global_step‚4999 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 _runtime‚2059.821859725 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88709668 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚190180539127824 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚192677235956736 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3516568792064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38033797 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22016664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12822624 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚187663826393600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚190180522088464 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 % _timestamp‚1.7845039057586548e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38033797 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚196193892770816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚196193787709440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3516568792064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184044836279296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50686016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12822365 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38033795 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50675873 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360Ž ªÙ'ŒÓO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38033797 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚196193892770816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚196193787709440 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3516568792064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184044836279296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50686016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12822365 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38033795 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50675873 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50686016 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88719813 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚193686210163802 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3505584887504 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚196193787709440 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38033795 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34839291 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 trainer/global_step‚4999 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 _runtime‚2059.821859725 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88709668 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚190180539127824 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚192677235956736 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3516568792064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38033797 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22016664 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12822624 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚187663826393600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚190180522088464 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 % _timestamp‚1.7845039057586548e+09“v勇2- _wandbruntime‚2060  _runtime‚2060ÇfÉQ ŽØM L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34845796 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38041118 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34846041 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚192714585284608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3619696772096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚1703S¾ÜLÒ9360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88737046 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88726887 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3517246901760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚184080531436544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚190217394156224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚190217411195584 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22020957 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3517246901760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚404 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50695928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚192714602323968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12825084 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3619872984064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7845039060307097e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3517334930944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50685771 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚187700429324288 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4999 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88726887 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚193723654995172 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38041118 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38041116 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚193723758183118 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38041116 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3517334930944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184080556340224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3506346987534 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚196231832186368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22020954 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88737046 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚192714602323968 _runtime‚2060.094071885 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚187700228208640̈ã&ÝM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚192714602323968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12825084 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3619872984064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7845039060307097e+09 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3517334930944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50685771 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚187700429324288 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4999 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚88726887 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚193723654995172 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38041118 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38041116 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚193723758183118 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38041116 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3517334930944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184080556340224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3506346987534 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚196231832186368 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22020954 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88737046 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚192714602323968 _runtime‚2060.094071885 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚187700228208640 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34845796 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38041118 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34846041 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚192714585284608 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3619696772096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88737046 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88726887 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3517246901760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚184080531436544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚190217394156224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚190217411195584 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22020957 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3517246901760 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚245 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚404 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50695928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0”‰ÇZP2- _wandbruntime‚2060  _runtime‚2060§=Þ:‘ÿ /loss/FRt2/detection_loss‚1.5237266350816574 /loss/FBt2/detection_loss‚0.5819449880786363 &loss/total_loss‚10.875849106644763 0loss/FRt2/assignment_loss‚1.0446311935840413 1loss/FBt1/assignment_loss‚0.07134812586480424 .loss/FBt1/detection_loss‚1.174757987938144 epoch‚6 $ _timestamp‚1.784503906031795e+09 0loss/FRt1/assignment_loss‚2.5959587694527295 1loss/SRqqt2/assignment_loss‚0.629038833679949 1loss/SRqqt2/detection_loss‚0.7003388323947637 2loss/FBt2/assignment_loss‚0.022651483763827265 .loss/FRt1/detection_loss‚0.601274697220863 1loss/SRqqt1/detection_loss‚1.1613296670900062 trainer/global_step‚4999 _runtime‚2060.094638275 _step‚405 2loss/SRqqt1/assignment_loss‚0.7688478924953409ý‘w( ’„ 0loss/FRt1/assignment_loss‚2.5959587694527295 1loss/SRqqt2/assignment_loss‚0.629038833679949 1loss/SRqqt2/detection_loss‚0.7003388323947637 2loss/FBt2/assignment_loss‚0.022651483763827265 .loss/FRt1/detection_loss‚0.601274697220863 1loss/SRqqt1/detection_loss‚1.1613296670900062 trainer/global_step‚4999 _runtime‚2060.094638275 _step‚405 2loss/SRqqt1/assignment_loss‚0.7688478924953409 /loss/FRt2/detection_loss‚1.5237266350816574 /loss/FBt2/detection_loss‚0.5819449880786363 &loss/total_loss‚10.875849106644763 0loss/FRt2/assignment_loss‚1.0446311935840413 1loss/FBt1/assignment_loss‚0.07134812586480424 .loss/FBt1/detection_loss‚1.174757987938144 epoch‚6 $ _timestamp‚1.784503906031795e+09•™,”Äœ“‚(:‘ ë¼õÒÔäù¿ gpu.0.gpu‚80.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚112.659' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.553000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ä¤ú#q”‚(:æ ë¼õҎ‹À# proc.memory.rssMB‚ 2865.73828125 cpu‚26.628915939378892) disk./.usagePercent‚76.15229499939889% disk./.usageGB‚1169.1281089782715 disk.nvme0c0n1.in‚ 87.6328125! disk.nvme1c1n1.in‚ 10.48828125 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0$ memory_percent‚98.80728721618652 network.recv‚611832 disk.nvme0n1.in‚ 87.6328125' disk.nvme1c1n1.out‚3002.01708984375 disk.nvme1n1.in‚ 10.48828125% disk.nvme1n1.out‚3002.01708984375 disk.md0.out‚ 54.48828125' proc.memory.availableMB‚ 195.4140625 proc.cpu.threads‚17 network.sent‚ 6.555665e+06' disk.nvme0c0n1.out‚13813.1572265625% disk.nvme0n1.out‚13813.1572265625 disk.dm-0.in‚ 98.12109375) proc.memory.percent‚17.49107837677002" disk.dm-0.out‚15357.7529296875" disk.dm-1.out‚1340.43310546875 disk.rbd0.out‚0š¹Pˆ2•- _wandbruntime‚2079  _runtime‚2079]DÿrÉ–Ã _step‚406 ' lr-AdamW/pg1‚0.00010891724874656806 ' lr-AdamW/pg2‚0.00010891724874656806 trainer/global_step‚5049 % _timestamp‚1.7845039255877771e+09 _runtime‚2079.650918196 rËÎ—È % _timestamp‚1.7845039255877771e+09 _runtime‚2079.650918196 _step‚406 ' lr-AdamW/pg1‚0.00010891724874656806 ' lr-AdamW/pg2‚0.00010891724874656806 trainer/global_step‚5049–Mö2˜- _wandbruntime‚2079  _runtime‚2079°Ï"Ô'™ÎO QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38399847 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 trainer/global_step‚5049 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51172625 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38399845 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12946086 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚192024142515824 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚198096043812864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚198096148874240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189483549292032 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚407 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚185829619045376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89572470 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚194545554318336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚185829594141696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35177519 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 195093504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚198096043812864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35177250 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3653955150336 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51182768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22231167 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38399847 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3539500511386 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚2079.655633836 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚194545554318336 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3654150243840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚195563643027210 % _timestamp‚1.7845039255887392e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚192024125476464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3550594555904 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚194545537278976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38399845 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482;vTÓšÓO N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 trainer/global_step‚5049 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51172625 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38399845 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12946086 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚192024142515824 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚198096043812864 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚198096148874240 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189483549292032 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚407 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚185829619045376 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚89572470 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚194545554318336 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚185829594141696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35177519 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 195093504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚198096043812864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35177250 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3653955150336 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51182768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22231167 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38399847 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3539500511386 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _runtime‚2079.655633836 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚194545554318336 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3654150243840 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚195563643027210 % _timestamp‚1.7845039255887392e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚192024125476464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3550594555904 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚194545537278976 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38399845 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38399847—l:…¼2›- _wandbruntime‚2079  _runtime‚2079ñƒÊÞ&œØM REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚89589689 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚185865289298944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚198134193358336 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35184231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3540176462942 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38407166 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189520155319296 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚5049 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚198134088289792 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚194582920685568 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89599848 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3551272672768 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚192060997544224 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚198134193358336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22235457 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3654656516096 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38407168 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38407166 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚194582920685568 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚194582903646208 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3551184643584 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35183975 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12948774 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38407168 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3654841116672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3540090314356 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51182523 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51192680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚194582903646208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89599848 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚2079.921901424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _step‚408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚198134193358336 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35184231 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3540176462942 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38407166 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚189520155319296 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚5049 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚198134088289792 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚194582920685568 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚89599848 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3551272672768 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚192060997544224 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚198134193358336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22235457 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3654656516096 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38407168 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38407166 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚194582920685568 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚194582903646208 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3551184643584 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35183975 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12948774 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38407168 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3654841116672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3540090314356 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51182523 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51192680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚194582903646208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚89599848 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚2079.921901424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _step‚408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199998299916288 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚197440972702784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚199998404977664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51669377 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22445664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚191303289666048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚196413855640576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚187614376907776 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3584532297728 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚196413855640576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199998299916288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90435272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38765895 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚196413872679936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚193867745903824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3573329986794 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚196413872679936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22445667 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3688937661952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35515309 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚187614352004096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13069645 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191303492886016¤ïòØ'¨ÒO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3584532297728 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚196413855640576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199998299916288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90435272 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38765895 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚196413872679936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚193867745903824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3573329986794 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚196413872679936 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22445667 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3688937661952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35515309 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚187614352004096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13069645 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191303492886016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚411 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38765897 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚193867728864464 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 trainer/global_step‚5099 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51669377 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51679520 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚197441075890618 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3584444275712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13069906 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199998299916288 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚197440972702784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚199998404977664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51669377 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22445664 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚191303289666048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚196413855640576 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚187614376907776 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104›ó!¹F2©- _wandbruntime‚2099  _runtime‚2099–*uÞ&ªØM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚35522249 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚196451239047168 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚196451222007808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3585122385408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22449954 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚193904600932224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3573919789764 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3585122385408 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚200036449461760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3689820235776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚197478520721988 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚191339682796544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚200036449461760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90462650 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚196451239047168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚412 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90452491 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51679275 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚191339892300800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13072030 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚193904617971584 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 % _timestamp‚1.7845039459354692e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38773216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38773218 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚187650072065024 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3585210414592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51679275 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51689432 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚197478623909934 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38773218 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3574005938350 _runtime‚2099.998832273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚196451222007808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚200036344393216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38773216 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51689432 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚187650047161344 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90462650 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13072292 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3689635635200 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90452491 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3585210414592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22449957 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35521984 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90462650 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚196451239047168 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚412 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚90452491 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51679275 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚191339892300800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.laoë|Urge_pool.current‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13072030 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚193904617971584 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 % _timestamp‚1.7845039459354692e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38773216 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38773218 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚187650072065024 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3585210414592 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51679275 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51689432 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚197478623909934 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38773218 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3574005938350 _runtime‚2099.998832273 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚196451222007808 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚200036344393216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38773216 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51689432 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚187650047161344 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90462650 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13072292 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3689635635200 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90452491 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3585210414592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22449957 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35521984 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3585122385408 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚200036449461760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3689820235776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚197478520721988œo¸™2¬- _wandbruntime‚2099  _runtime‚2099þ‘N­ÿ _runtime‚2099.999415483 _step‚413 /loss/FRt1/assignment_loss‚2.625702175951789 0loss/FRt2/assignment_loss‚0.9649634822405906 epoch‚6 trainer/global_step‚5099 % _timestamp‚1.7845039459365785e+09 2loss/SRqqt2/assignment_loss‚0.6584262894907927 1loss/SRqqt1/detection_loss‚1.3715533547778138 /loss/FBt2/detection_loss‚0.5040133994269227 1loss/FBt1/assignment_loss‚0.07127700304008613 1loss/FBt2/assignment_loss‚0.01679534241482458 1loss/SRqqt2/detection_loss‚0.7268315479353381 &loss/total_loss‚11.503269972591792 2loss/SRqqt1/assignment_loss‚0.8919692040154976 .loss/FRt1/detection_loss‚0.593266496321207 /loss/FRt2/detection_loss‚1.6011585538302369 .loss/FBt1/detection_loss‚1.477313123146693,7 ®„ _runtime‚2099.999415483 _step‚413 /loss/FRt1/assignment_loss‚2.625702175951789 0loss/FRt2/assignment_loss‚0.9649634822405906 epoch‚6 trainer/global_step‚5099 % _timestamp‚1.7845039459365785e+09 2loss/SRqqt2/assignment_loss‚0.6584262894907927 1loss/SRqqt1/detection_loss‚1.3715533547778138 /loss/FBt2/detection_loss‚0.5040133994269227 1loss/FBt1/assignment_loss‚0.07127700304008613 1loss/FBt2/assignment_loss‚0.01679534241482458 1loss/SRqqt2/detection_loss‚0.7268315479353381 &loss/total_loss‚11.503269972591792 2loss/SRqqt1/assignment_loss‚0.8919692040154976 .loss/FRt1/detection_loss‚0.593266496321207 /loss/FRt2/detection_loss‚1.6011585538302369 .loss/FBt1/detection_loss‚1.477313123146693“²Ò¯o¯‚(:ä ‰½õÒêæõ¿ proc.cpu.threads‚17 network.sent‚ 6.681608e+06 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 87.6328125% disk.nvme1n1.out‚3010.21533203125) disk./.usagePercent‚76.04999029758679% disk./.usageGB‚1167.5574760437012! disk.dm-1.out‚1341.2177734375( disk.nvme0c0n1.out‚13857.27099609375! disk.nvme1c1n1.in‚ 10.48828125 disk.md0.out‚ 57.75390625 disk.dm-0.out‚ 15402.6015625( proc.memory.availableMB‚ 194.87890625 network.recv‚619810 disk.rbd0.in‚0 disk.nvme0n1.in‚ 87.6328125' disk.nvme1c1n1.out‚3010.21533203125 disk.nvme1n1.in‚ 10.48828125 disk.dm-0.in‚ 98.12109375$ memory_percent‚98.81055355072021& disk.nvme0n1.out‚13857.27099609375 disk.md0.in‚0# proc.memory.rssMB‚ 2865.73828125) proc.memory.percent‚17.49107837677002 cpu‚26.61845830066715ÁNseœ°‚(:‘ ‰½õÒøáœÀ gpu.0.gpu‚52.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚189.097' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚63.032333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•’òøg±‚(:Ü ˜½õÒ Èå¿ disk.rbd0.in‚0 disk.rbd0.out‚0 cpu‚26.613517269229316# disk./.usageGB‚1167.55757522583 disk.nvme0n1.in‚ 87.6328125% disk.nvme0n1.out‚13858.6123046875% disk.nvme1c1n1.out‚3010.990234375# disk.nvme1n1.out‚3010.990234375 disk.dm-0.in‚ 98.12109375# proc.memory.rssMB‚ 2865.73828125 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 87.6328125' disk.nvme0c0n1.out‚13858.6123046875 disk.dm-0.out‚15403.40234375 disk.dm-1.in‚0! disk.dm-1.out‚1341.3193359375( proc.memory.availableMB‚ 195.01171875 network.recv‚623834 disk.md0.in‚0 disk.md0.out‚ 58.3203125$ memory_percent‚98.80974292755127) proc.memory.percent‚17.49107837677002 network.sent‚ 6.729259e+06( disk./.usagePercent‚76.0499967579111! disk.nvme1c1n1.in‚ 10.48828125 disk.nvme1n1.in‚ 10.48828125Öýœ²‚(:‘ ˜½õÒ‹Ä¥À gpu.0.gpu‚89.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚196.652' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.55066666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~EáA2³- _wandbruntime‚2119  _runtime‚2119iNÑ–É´Ã ' lr-AdamW/pg1‚0.00010876589700449798 ' lr-AdamW/pg2‚0.00010876589700449798 trainer/global_step‚5149 % _timestamp‚1.7845039657154818e+09 _runtime‚2119.778663828 _step‚414‰ÑrÎµÈ _runtime‚2119.778663828 _step‚414 ' lr-AdamW/pg1‚0.00010876589700449798 ' lr-AdamW/pg2‚0.00010876589700449798 trainer/global_step‚5149 % _timestamp‚1.7845039657154818e+09žâÜJž2¶- _wandbruntime‚2119  _runtime‚2119U íÓ'·ÍO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13193276 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39131947 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚199318405566192 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39131945 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7845039657164836e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3618470039552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚189399134770176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3618382017536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚198282174002176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚5149 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52176272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚189399109866496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198282174002176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22660167 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201900556019712 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚193123017325568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35853715 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚201900661081088 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91308219 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3618382017536 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚201900556019712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13193548 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198282191041536 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3618470039552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52176272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3723907459072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3607159462202 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52166129 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚201900661081088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22660164 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3724094163968 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13193548 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198282191041536 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3618470039552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52176272 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3723907459072 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3607159462202 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52166129 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚201900661081088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22660164 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3724094163968 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3618470039552 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚189399134770176 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3618382017536 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚198282174002176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚5149 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52176272 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚189399109866496 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚198282174002176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22660167 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚201900556019712 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚193123017325568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35853715 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚201900661081088 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91308219 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3618382017536 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834Ÿwô”ô2¹- _wandbruntime‚2120  _runtime‚2120<]º×M UHDeviceStat‘^­ÙÉ&sMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91325452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3724792208896 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91315293 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚199356056773342 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3619060127232 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 trainer/global_step‚5149 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35860141 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39139266 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3619148156416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3619148156416 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193159622136320 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189434829927424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚199355953585396 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚201938705565184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚198319540369408 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚195748204320224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3607835413758 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52176027 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚195748221359584 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚198319540369408 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52176027 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚201938705565184 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91325452 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚2120.051445443 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91315293 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚198319557408768 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189434805023744 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚198319557408768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193159410534912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 $ _timestamp‚1.784503965988117e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚201938600496640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13195687 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946Ÿ"â&»ÜM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚198319540369408 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52176027 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚201938705565184 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91325452 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚2120.051445443 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91315293 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚198319557408768 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189434805023744 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚198319557408768 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚193159410534912 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 $ _timestamp‚1.784503965988117e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚201938600496640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13195687 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚91325452 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3724792208896 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91315293 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚199356056773342 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3619060127232 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 trainer/global_step‚5149 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35860141 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39139266 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3619148156416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3619148156416 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚193159622136320 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189434829927424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚199355953585396 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚201938705565184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚198319540369408 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚195748204320224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3607835413758 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52176027 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚195748221359584 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ëm?+2¼- _wandbruntime‚2120  _runtime‚2120vïßV½þ /loss/FRt1/assignment_loss‚2.600284418457537 .loss/FRt2/detection_loss‚1.432921210621648 0loss/SRqqt2/detection_loss‚0.680854363887213 &loss/total_loss‚11.016903277633228 trainer/global_step‚5149 2loss/SRqqt2/assignment_loss‚0.6453542446150013 /loss/FRt1/detection_loss‚0.5801944009521995 epoch‚6 0loss/FRt2/assignment_loss‚0.9821742791755527 % _timestamp‚1.7845039659891667e+09 _runtime‚2120.052077515 2loss/SRqqt1/assignment_loss‚0.6157166025746469 1loss/FBt1/assignment_loss‚0.08780633446440334 1loss/FBt2/assignment_loss‚0.04846253381502111 1loss/SRqqt1/detection_loss‚1.1328526888187829 .loss/FBt1/detection_loss‚1.344441409662573 /loss/FBt2/detection_loss‚0.8658407905886489 _step‚417a†d ¾ƒ 0loss/FRt2/assignment_loss‚0.9821742791755527 % _timestamp‚1.7845039659891667e+09 _runtime‚2120.052077515 2loss/SRqqt1/assignment_loss‚0.6157166025746469 1loss/FBt1/assignment_loss‚0.08780633446440334 1loss/FBt2/assignment_loss‚0.04846253381502111 1loss/SRqqt1/detection_loss‚1.1328526888187829 .loss/FBt1/detection_loss‚1.344441409662573 /loss/FBt2/detection_loss‚0.8658407905886489 _step‚417 /loss/FRt1/assignment_loss‚2.600284418457537 .loss/FRt2/detection_loss‚1.432921210621648 0loss/SRqqt2/detection_loss‚0.680854363887213 &loss/total_loss‚11.016903277633228 trainer/global_step‚5149 2loss/SRqqt2/assignment_loss‚0.6453542446150013 /loss/FRt1/detection_loss‚0.5801944009521995 epoch‚6¡¯Þ¿‚(:… §½õÒ»î¿ gpu.0.gpu‚41.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚180.477' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚60.159 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çŒMçmÀ‚(:â §½õÒ°ÿþ¿% disk./.usageGB‚1167.5575828552246 disk.nvme0n1.in‚ 87.6328125 disk.nvme1n1.in‚ 10.48828125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 87.6328125& proc.memory.availableMB‚ 194.546875) disk./.usagePercent‚76.04999725485911 disk.dm-0.in‚ 98.12109375( disk.nvme0c0n1.out‚13867.82177734375) proc.memory.percent‚17.49107837677002 proc.cpu.threads‚17 network.sent‚ 6.748007e+06 disk.md0.out‚ 59.02734375 disk.dm-1.out‚ 1341.4453125# proc.memory.rssMB‚ 2865.73828125 cpu‚26.608403635678343& disk.nvme0n1.out‚13867.82177734375! disk.nvme1c1n1.in‚ 10.48828125& disk.nvme1c1n1.out‚3012.4326171875$ disk.nvme1n1.out‚3012.4326171875 disk.md0.in‚0# disk.dm-0.out‚15412.44384765625 disk.dm-1.in‚0$ memory_percent‚98.81258010864258 network.recv‚626981g/2Á- _wandbruntime‚2139  _runtime‚2139[ xÉÂà ' lr-AdamW/pg1‚0.00010868895432043167 ' lr-AdamW/pg2‚0.00010868895432043167 trainer/global_step‚5199 % _timestamp‚1.7845039855188737e+09 _runtime‚2139.582029801 _step‚418ÕjA´ÎÃÈ ' lr-AdamW/pg1‚0.00010868895432043167 ' lr-AdamW/pg2‚0.00010868895432043167 trainer/global_step‚5199 % _timestamp‚1.7845039855188737e+09 _runtime‚2139.582029801 _step‚418¢‹þ¤ð2Ä- _wandbruntime‚2139  _runtime‚2139¿”Ê$ÅÎO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚201195838429600 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚200150492363776 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3652319759360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22874664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3652407781376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3758877308416 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚92160876 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚191183892632576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52673024 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _step‚419 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚197554935640464 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3640902789136 % _timestamp‚1.7845039855198326e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52673024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13317866 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚5199 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3652319759360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22874667 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39497997 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13317624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚191183867728896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚203802917184512 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚92171021 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚203802812123136 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚200150509403136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚194942948257280 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3759055624704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3640988937610 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36192533 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚203802917184512 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚203802812123136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39497997 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52662881 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36192288 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 _runtime‚2139.586624319 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚194942745037312 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚197554952679824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚203802812123136 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚200150509403136 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚194942948257280 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3759055624704 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3640988937610 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36192533 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚203802917184512 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚203802812123136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39497997 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52662881 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36192288 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 _runtime‚2139.586624319 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚194942745037312 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚197554952679824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚197554935640464 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3640902789136 % _timestamp‚1.7845039855198326e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52673024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13317866 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚5199 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3652319759360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22874667 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39497997 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13317624 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚191183867728896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152£ÿ‰=º2Ç- _wandbruntime‚2139  _runtime‚2139LdhÝ&È×M FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92178095 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚197591824747584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚200187858731008 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92188254 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52682936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52682936 trainer/global_step‚5199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52672779 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚203840961668608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13320052 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3641578740580 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚200187858731008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36199006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22878957 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚201233386448804 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚194979135944704 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92178095 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚200187875770368 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _step‚420 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3652997869056 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13320302 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22878954 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚203840856600064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚191219562886144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39505318 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚197591807708224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3759759756288 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92188254 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39505318 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36199259 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚200187875770368¨É‘:â&ÉÜM VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚194979135944704 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚92178095 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚200187875770368 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _step‚420 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3652997869056 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13320302 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22878954 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚203840856600064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚191219562886144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39505318 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚197591807708224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3759759756288 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚92188254 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39505318 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36199259 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚200187875770368 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚92178095 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚197591824747584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚200187858731008 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚92188254 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚52682936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52682936 trainer/global_step‚5199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52672779 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚203840961668608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13320052 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3641578740580 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚200187858731008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36199006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22878957 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚201233386448804 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104¤}}qÄ2Ê- _wandbruntime‚2139  _runtime‚2139|“ÈËÿ 1loss/SRqqt2/detection_loss‚0.8046517758053352 &loss/total_loss‚11.520064774834458 $ _timestamp‚1.784503985786872e+09 0loss/FBt1/assignment_loss‚0.2135537860980766 /loss/FRt1/detection_loss‚0.5800782689960519 /loss/FBt1/detection_loss‚1.3161416157432224 trainer/global_step‚5199 0loss/FRt2/assignment_loss‚1.0216464672210444 /loss/FBt2/detection_loss‚0.5583061437074334 _runtime‚2139.849847531 _step‚421 2loss/SRqqt1/assignment_loss‚0.9657808204324313 0loss/SRqqt1/detection_loss‚1.319964101581236 epoch‚6 0loss/FRt1/assignment_loss‚2.5161423459955694 2loss/SRqqt2/assignment_loss‚0.6805505267273176 1loss/FBt2/assignment_loss‚0.04936879983067039 /loss/FRt2/detection_loss‚1.4938801226960698qN Ì„ epoch‚6 0loss/FRt1/assignment_loss‚2.5161423459955694 2loss/SRqqt2/assignment_loss‚0.6805505267273176 1loss/FBt2/assignment_loss‚0.04936879983067039 /loss/FRt2/detection_loss‚1.4938801226960698 1loss/SRqqt2/detection_loss‚0.8046517758053352 &loss/total_loss‚11.520064774834458 $ _timestamp‚1.784503985786872e+09 0loss/FBt1/assignment_loss‚0.2135537860980766 /loss/FRt1/detection_loss‚0.5800782689960519 /loss/FBt1/detection_loss‚1.3161416157432224 trainer/global_step‚5199 0loss/FRt2/assignment_loss‚1.0216464672210444 /loss/FBt2/detection_loss‚0.5583061437074334 _runtime‚2139.849847531 _step‚421 2loss/SRqqt1/assignment_loss‚0.9657808204324313 0loss/SRqqt1/detection_loss‚1.319964101581236¥Í'N`¬Í‚(:Ø ¶½õÒî”í¿ disk.nvme0n1.in‚ 87.6328125 disk.nvme1n1.in‚ 10.48828125 disk.nvme1n1.out‚ 3018.109375 disk.md0.out‚ 62.953125 disk.rbd0.out‚¶™;°·0 disk.nvme0c0n1.in‚ 87.6328125) proc.memory.percent‚17.49107837677002 network.recv‚631317 disk.dm-0.in‚ 98.12109375 disk.dm-0.out‚15427.07421875! disk.dm-1.out‚1341.7060546875$ memory_percent‚98.81279468536377( proc.memory.availableMB‚ 194.51171875# proc.memory.rssMB‚ 2865.73828125 network.sent‚ 6.796479e+06) disk./.usagePercent‚76.01902322997451! disk.nvme1c1n1.in‚ 10.48828125" disk.nvme1c1n1.out‚ 3018.109375 disk.md0.in‚0 cpu‚26.603601412885727% disk./.usageGB‚1167.0820541381836% disk.nvme0n1.out‚13884.9619140625 disk.dm-1.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚13884.9619140625 proc.cpu.threads‚17õ)µÚ™Î‚(:Ž ¶½õÒ»æÚ¿ gpu.0.gpu‚55.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚131.2' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚43.73333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qMÝ©2Ï- _wandbruntime‚2158  _runtime‚2158ÐùxsÈР$ _timestamp‚1.784504004764439e+09 _runtime‚2158.827580264 _step‚422 ' lr-AdamW/pg1‚0.00010861116873302864 ' lr-AdamW/pg2‚0.00010861116873302864 trainer/global_step‚5249°×S§ÍÑÇ ' lr-AdamW/pg1‚0.00010861116873302864 ' lr-AdamW/pg2‚0.00010861116873302864 trainer/global_step‚5249 $ _timestamp‚1.784504004764439e+09 _runtime‚2158.827580264 _step‚422¦Že.O2Ò- _wandbruntime‚2158  _runtime‚2158céÔ'ÓÎO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚202018827764736 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3793971409408 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3793793093120 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53169776 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53169776 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36530141 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93033823 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚192968625591296 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39864047 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚202018827764736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚196762621904384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚199398556067824 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3674818413018 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13440974 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚203073271293008 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚199398539028464 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚196762418684416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚205705173287936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13440714 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53159633 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3674732264544 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚205705068226560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3686345523200 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3686257501184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39864045 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53159633 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39864047 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚2158.828441275 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚205705173287936 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _step‚423 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39864045 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚202018810725376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3686345523200 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚205705068226560 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23089164 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚192968650494976 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36529878 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 % _timestamp‚1.7845040047653778e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚202018810725376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚203073374480842 trainer/global_step‚5249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23089167 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚196762418684416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚205705173287936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13440714 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53159633 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3674732264544 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚205705068226560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3686345523200 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3686257501184 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚39864045 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53159633 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39864047 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚2158.828441275 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚205705173287936 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _step‚423 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39864045 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚202018810725376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3686345523200 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚205705068226560 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23089164 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚192968650494976 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36529878 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 % _timestamp‚1.7845040047653778e+09 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚202018810725376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚203073374480842 trainer/global_step‚5249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23089167 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53169776 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36530141 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93033823 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚192968625591296 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39864047 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚202018827764736 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚196762621904384 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚199398556067824§¥¯$G2Õ- _wandbruntime‚2159  _runtime‚2159…æþÞ&ÖØM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚193004345652224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚202056177092608 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93040897 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚199435411096224 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚202056177092608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚193004320748544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚203110819312212 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93051056 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚5249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 % _timestamp‚1.7845040053459342e+09 _runtime‚2159.409241324 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36536570 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93040897 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚196799019504640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23093454 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53179688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚205743217772032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚424 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚202056194131968 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3686935610880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13443357 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13443116 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3675408215988 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3686935610880 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53179688 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39871368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23093457 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3794673852416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39871368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚205743112703488 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39871366 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚202056194131968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39871366 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚196798814194688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚203110922500158 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚199435428135584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3675494364574 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53169531 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚205743217772032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3687023640064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53169531 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93051056 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.alloca¥•‰r6$ted‚199435428135584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3675494364574 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53169531 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚205743217772032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3687023640064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53169531 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93051056 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚202056177092608 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93040897 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚199435411096224 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚202056177092608 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚193004320748544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚203110819312212 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93051056 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚5249 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 % _timestamp‚1.7845040053459342e+09 _runtime‚2159.409241324 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36536570 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93040897 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚196799019504640 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23093454 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚53179688 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚205743217772032 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚424 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚202056194131968 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3686935610880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13443357 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13443116 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3675408215988 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3686935610880 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53179688 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39871368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23093457 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3794673852416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39871368 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚205743112703488 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39871366 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚202056194131968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39871366 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚196798814194688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚203110922500158 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0¨'[h92Ø- _wandbruntime‚2159  _runtime‚2159‚wÐ\Ù€ /loss/FRt2/detection_loss‚1.4257396000689886 &loss/total_loss‚10.444064871598222 trainer/global_step‚5249 /loss/FRt1/assignment_loss‚2.412312029100401 2loss/SRqqt1/assignment_loss‚0.7658862003155293 epoch‚6 _runtime‚2159.410313566 0loss/FRt2/assignment_loss‚1.0209978762127103 2loss/SRqqt2/assignment_loss‚0.6713654887880388 0loss/FBt1/assignment_loss‚0.1425360601600665 /loss/FRt1/detection_loss‚0.6024339235406869 1loss/SRqqt1/detection_loss‚1.0896652455591866 1loss/SRqqt2/detection_loss‚0.7548982464310364 /loss/FBt2/detection_loss‚0.5442332348627446 _step‚425 1loss/FBt2/assignment_loss‚0.03797327478081355 /loss/FBt1/detection_loss‚0.9760236917780177 % _timestamp‚1.7845040053470206e+09’¼¾' Ú… /loss/FBt2/detection_loss‚0.5442332348627446 _step‚425 1loss/FBt2/assignment_loss‚0.03797327478081355 /loss/FBt1/detection_loss‚0.9760236917780177 % _timestamp‚1.7845040053470206e+09 /loss/FRt2/detection_loss‚1.4257396000689886 &loss/total_loss‚10.444064871598222 trainer/global_step‚5249 /loss/FRt1/assignment_loss‚2.412312029100401 2loss/SRqqt1/assignment_loss‚0.7658862003155293 epoch‚6 _runtime‚2159.410313566 0loss/FRt2/assignment_loss‚1.0209978762127103 2loss/SRqqt2/assignment_loss‚0.6713654887880388 0loss/FBt1/assignment_loss‚0.1425360601600665 /loss/FRt1/detection_loss‚0.6024339235406869 1loss/SRqqt1/detection_loss‚1.0896652455591866 1loss/SRqqt2/detection_loss‚0.7548982464310364©Œ\›Û‚(: ŽõÒÚ™î¿ gpu.0.gpu‚57.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚176.462' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚58.82066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óéª)`Ü‚(:Õ ŽõÒå·äÀ network.sent‚ 6.858814e+06! disk.nvme1c1n1.in‚ 10.48828125 disk.nvme1n1.in‚ 10.48828125 disk.md0.in‚0 disk.dm-1.in‚0 cpu‚26.598723043815102 proc.cpu.threads‚17 disk.nvme0c0n1.in‚90.21875& disk.nvme0n1.out‚13922.38037109375 disk.nvme1n1.out‚ 3022.28125 disk.md0.out‚ 63.4921875! disk.dm-1.out‚1342.1337890625$ memory_percent‚98.81458282470703) disk./.usagePercent‚76.01901155169595$ disk./.usageGB‚1167.081874847412 disk.rbd0.out‚0 disk.nvme0n1.in‚90.21875! disk.nvme1c1n1.out‚ 3022.28125 disk.dm-0.in‚ 100.70703125# disk.dm-0.out‚15467.10009765625% proc.memory.availableMB‚ 194.21875 network.recv‚635325 disk.rbd0.in‚0( disk.nvme0c0n1.out‚13922.38037109375# proc.memory.rssMB‚ 2865.73828125) proc.memory.percent‚17.49107837677002[R½-iÝ‚(:Þ Ô½õÒÀàë¿( proc.memory.availableMB‚ 194.69140625 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚91.03125$ disk.nvme1n1.out‚3023.0439453125 network.recv‚639481( disk./.usagePercent‚76.0190190059163# disk./.usageGB‚1167.08198928833 disk.dm-0.in‚ 101.51953125# disk.dm-0.out‚15472.85302734375! disk.dm-1.out‚1342.2587890625& disk.nvme1c1n1.out‚3023.0439453125) proc.memory.percent‚17.49107837677002 network.sent‚ 6.922987e+06( disk.nvme0c0n1.out‚13928.65380859375& disk.nvme0n1.out‚13928.65380859375# memory_percent‚98.8116979598999# proc.memory.rssMB‚ 2865.73828125 cpu‚26.59407341499039 disk.rbd0.in‚0 disk.md0.in‚0 disk.md0.out‚ 64.0390625 disk.nvme0n1.in‚91.03125! disk.nvme1c1n1.in‚ 10.48828125 disk.nvme1n1.in‚ 10.48828125æIQ¨›Þ‚(: Ô½õÒÉŸ–À gpu.0.gpu‚63.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚196.847' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.61566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óòç2ß- _wandbruntime‚2178  _runtime‚2178 pw*Éàà ' lr-AdamW/pg2‚0.00010853254148818301 trainer/global_step‚5299 % _timestamp‚1.7845040243486688e+09 _runtime‚2178.411808947 _step‚426 ' lr-AdamW/pg1‚0.00010853254148818301ŠlpÎáÈ % _timestamp‚1.7845040243486688e+09 _runtime‚2178.411808947 _step‚426 ' lr-AdamW/pg1‚0.00010853254148818301 ' lr-AdamW/pg2‚0.00010853254148818301 trainer/global_step‚5299ª·dmë2â- _wandbruntime‚2178  _runtime‚2178ô×±Ô'ãÎO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40230095 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36868302 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚201242142416464 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚203887146126336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198582130379264 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40230097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194753383453696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3720283265024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93886480 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207607324329984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚204950704156416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3720283265024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3828746925568 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3720195243008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198582333599232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚204950807344250 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 trainer/global_step‚5299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3720195243008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23303664 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40230097 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36868055 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3708561739952 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚207607429391360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13564391 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚203887146126336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53666528 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53666528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚203887146126336 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198582130379264 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40230097 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194753383453696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3720283265024 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93886480 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚207607324329984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚204950704156416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3720283265024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3828746925568 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3720195243008 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚198582333599232 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚204950807344250 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 trainer/global_step‚5299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3720195243008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23303664 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40230097 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36868055 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3708561739952 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚207607429391360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13564391 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚203887146126336 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53666528 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚53666528 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚204988252175620 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚201279031523584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚207645473875456 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚203924512493568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚198618738667520 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3829635152896 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40237418 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40237416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13567102 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40237418 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93903699 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 _runtime‚2178.679773247 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3720961381888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23307954 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23307957 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3720961381888 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53666283 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚203924495454208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 197183488 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 _step‚428 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚204988355363566 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53666283 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36874802 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198618516580352 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93913858 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚203924495454208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚203924512493568 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93913858 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93903699 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 trainer/global_step‚5299 % _timestamp‚1.7845040246164365e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53676440 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3829437969408 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚201279014484224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207645368806912 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36875059 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330$ußÇã&çÝM trainer/global_step‚5299 % _timestamp‚1.7845040246164365e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53676440 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3829437969408 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚201279014484224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚207645368806912 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚36875059 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40237416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚204988252175620 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚201279031523584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚207645473875456 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚203924512493568 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚198618738667520 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3829635152896 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40237418 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40237416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13567102 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40237418 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93903699 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 _runtime‚2178.679773247 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3720961381888 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23307954 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23307957 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3720961381888 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53666283 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚203924495454208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 197183488 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 _step‚428 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚204988355363566 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53666283 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36874802 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198618516580352 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93913858 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚203924495454208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚203924512493568 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93913858 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93903699 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586¬ÎQK2è- _wandbruntime‚2178  _runtime‚2178â ý¼é‚ &loss/total_loss‚10.571614450342476 % _timestamp‚1.7845040246174881e+09 _runtime‚2178.680451479 0loss/FRt1/assignment_loss‚2.6182500712349692 2loss/SRqqt2/assignment_loss‚0.5734238998119138 1loss/FBt1/assignment_loss‚0.16127473856968558 epoch‚6 trainer/global_step‚5299 2loss/SRqqt1/assignment_loss‚0.7151695896381847 1loss/FBt2/assignment_loss‚0.05355826393626941 /loss/FRt2/detection_loss‚1.4827267933271866 /loss/FBt1/detection_loss‚0.8851883941544936 /loss/FBt2/detection_loss‚0.5769466733566957 _step‚429 0loss/FRt2/assignment_loss‚1.0676996566990549 /loss/FRt1/detection_loss‚0.5958424659737295 1loss/SRqqt1/detection_loss‚1.1072871139249167 1loss/SRqqt2/detection_loss‚0.7342467897153757 ÓDW ê‡ 2loss/SRqqt1/assignment_loss‚0.7151695896381847 1loss/FBt2/assignment_loss‚0.05355826393626941 /loss/FRt2/detection_loss‚1.4827267933271866 /loss/FBt1/detection_loss‚0.8851883941544936 /loss/FBt2/detection_loss‚0.5769466733566957 _step‚429 0loss/FRt2/assignment_loss‚1.0676996566990549 /loss/FRt1/detection_loss‚0.5958424659737295 1loss/SRqqt1/detection_loss‚1.1072871139249167 1loss/SRqqt2/detection_loss‚0.7342467897153757 &loss/total_loss‚10.571614450342476 % _timestamp‚1.7845040246174881e+09 _runtime‚2178.680451479 0loss/FRt1/assignment_loss‚2.6182500712349692 2loss/SRqqt2/assignment_loss‚0.5734238998119138 1loss/FBt1/assignment_loss‚0.16127473856968558 epoch‚6 trainer/global_step‚5299­~²<†në‚(:ã ã½õÒ¤®•À( disk.nvme0c0n1.out‚13941.39794921875& disk.nvme0n1.out‚13941.39794921875! disk.nvme1c1n1.in‚ 10.48828125% disk.nvme1n1.out‚3025.76416015625# disk.dm-0.out‚15485.65966796875# proc.memory.rssMB‚ 2865.73828125 network.recv‚643048 disk.nvme0c0n1.in‚91.03125 disk.nvme1n1.in‚ 10.48828125% proc.memory.availableMB‚ 194.65625 cpu‚26.589455001040832) disk./.usagePercent‚75.84506309570153 disk.dm-1.in‚0" disk.dm-1.out‚1342.58837890625 disk.nvme0n1.in‚91.03125' disk.nvme1c1n1.out‚3025.76416015625 disk.md0.out‚ 65.16796875) proc.memory.percent‚17.49107837677002 proc.cpu.threads‚17 network.sent‚ 6.938895e+06 disk.rbd0.out‚0 disk.md0.in‚0 disk.dm-0.in‚ 101.51953125# memory_percent‚98.8119125366211% disk./.usageGB‚1164.4113311767578 disk.rbd0.in‚0×É—Ç›ì‚(: ã½õÒÙ¿ÅÀ gpu.0.gpu‚59.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚193.682' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚64.56066666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lé¬2í- _wandbruntime‚2198  _runtime‚2198!$ÓôÉîà ' lr-AdamW/pg1‚0.00010845307384526965 ' lr-AdamW/pg2‚0.00010845307384526965 trainer/global_step‚5349 % _timestamp‚1.7845040439485931e+09 _runtime‚2198.011622315 _step‚430$m§LÎïÈ ' lr-AdamW/pg1‚0.00010845307384526965 ' lr-AdamW/pg2‚0.00010845307384526965 trainer/global_step‚5349 % _timestamp‚1.7845040439485931e+09 _runtime‚2198.011622315 _step‚430®“Á_û2ð- _wandbruntime‚2198  _runtime‚2198c=o›ÞñÎO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54153137 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54153137 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚206828137019824 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚205755447448576 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3754221006848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3742477363834 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚209509580433408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94759427 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3863868611584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚203085762843824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3754132984832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13688738 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚209509685494784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54163280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192996352 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚203085745804464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _runtime‚2198.013527919 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3754221006848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40596147 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217900032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3754132984832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23518167 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200401816931328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚431 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40596145 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚205755447448576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40596147 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37206905 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37206643 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚206828240207658 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3742391215360 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚209509580433408 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94759427 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3863868611584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚203085762843824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3754132984832 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13688738 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚209509685494784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54163280 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192996352 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚203085745804464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _runtime‚2198.013527919 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3754221006848 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40596147 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217900032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3754132984832 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23518167 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200401816931328 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚431 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚40596145 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚205755447448576 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40596147 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37206905 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37206643 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚206828240207658 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3742391215360 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54163035 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚200438423024128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54173192 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚209547624910336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚203122617872224 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _runtime‚2198.278661932 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚432 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚203122634911584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40603468 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94766501 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54173192 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3743067166804 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37213679 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚205792813815808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3754811094528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚205792813815808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200438215617024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54163035 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94776660 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚209547729978880 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3754811094528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚209547729978880 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37213417 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3864561647104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3864379143680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94776660 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23522457 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23522454 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3743153315390 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104]¹R]ã&õÝM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37213679 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚205792813815808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3754811094528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚205792813815808 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚200438215617024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54163035 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚94776660 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚209547729978880 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3754811094528 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚209547729978880 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37213417 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3864561647104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3864379143680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚94776660 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23522457 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23522454 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3743153315390 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚54163035 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚200438423024128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54173192 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚209547624910336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚203122617872224 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _runtime‚2198.278661932 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚432 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚203122634911584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40603468 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94766501 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54173192 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3743067166804 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238°W}ž8ö- _wandbrunð,C1time‚2198  _runtime‚21987ˆ,‡÷ÿ 1loss/SRqqt2/assignment_loss‚0.557193613996273 0loss/FBt1/assignment_loss‚0.0514349943756743 1loss/FBt2/assignment_loss‚0.00592576048073236 1loss/SRqqt1/detection_loss‚1.1461726572941726 %loss/total_loss‚10.60738526270033 _runtime‚2198.279203573 0loss/FRt2/assignment_loss‚1.2263087698391506 2loss/SRqqt1/assignment_loss‚0.8313244266376476 /loss/FRt1/detection_loss‚0.6074719928565449 /loss/FRt2/detection_loss‚1.4905065258983903 /loss/FBt1/detection_loss‚0.8957739392952884 trainer/global_step‚5349 0loss/FRt1/assignment_loss‚2.4689478916689334 1loss/SRqqt2/detection_loss‚0.6314253257190682 /loss/FBt2/detection_loss‚0.6948993646384527 epoch‚6 % _timestamp‚1.7845040442163658e+09 _step‚433ÉõS ø„ 0loss/FRt2/assignment_loss‚1.2263087698391506 2loss/SRqqt1/assignment_loss‚0.8313244266376476 /loss/FRt1/detection_loss‚0.6074719928565449 /loss/FRt2/detection_loss‚1.4905065258983903 /loss/FBt1/detection_loss‚0.8957739392952884 trainer/global_step‚5349 0loss/FRt1/assignment_loss‚2.4689478916689334 1loss/SRqqt2/detection_loss‚0.6314253257190682 /loss/FBt2/detection_loss‚0.6948993646384527 epoch‚6 % _timestamp‚1.7845040442163658e+09 _step‚433 1loss/SRqqt2/assignment_loss‚0.557193613996273 0loss/FBt1/assignment_loss‚0.0514349943756743 1loss/FBt2/assignment_loss‚0.00592576048073236 1loss/SRqqt1/detection_loss‚1.1461726572941726 %loss/total_loss‚10.60738526270033 _runtime‚2198.279203573±Ï='À`ù‚(:Õ ò½õÒÏ¡‹À disk.dm-0.in‚ 101.51953125 disk.dm-0.out‚15506.33203125" disk.dm-1.out‚1342.69775390625 disk.rbd0.in‚0 disk.nvme0n1.in‚91.03125" disk.nvme0n1.out‚ 13962.4453125% disk.nvme1n1.out‚3026.33447265625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚91.03125$ disk.nvme0c0n1.out‚ 13962.4453125' disk.nvme1c1n1.out‚3026.33447265625 disk.md0.in‚0 disk.md0.out‚ 65.5546875# memory_percent‚99.2067813873291 network.sent‚ 7.00131e+06( disk./.usagePercent‚75.8450183703794 disk.dm-1.in‚0! disk.nvme1c1n1.in‚ 10.48828125' proc.memory.availableMB‚ 129.9609375 network.recv‚647504 disk.nvme1n1.in‚ 10.48828125# proc.memory.rssMB‚ 2865.73828125) proc.memory.percent‚17.49107837677002 cpu‚26.584909189857825 proc.cpu.threads‚17# disk./.usageGB‚1164.41064453125Ê^ˆñú‚(:’ ò½õÒ°œ¶À gpu.0.gpu‚89.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚137.966' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚45.988666666666674 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.&*2û- _wandbruntime‚2217  _runtime‚2217”jwÓÈü trainer/global_step‚5399 % _timestamp‚1.7845040635517666e+09 _runtime‚ 2217.61658604 _step‚434 ' lr-AdamW/pg1‚0.00010837276707712419 ' lr-AdamW/pg2‚0.00010837276707712419›8„$ÍýÇ ' lr-AdamW/pg1‚0.00010837276707712419 ' lr-AdamW/pg2‚0.00010837276707712419 trainer/global_step‚5399 % _timestamp‚1.7845040635517666e+09 _runtime‚ 2217.61658604 _step‚434²²¿¿õ2þ- _wandbruntime‚2217  _runtime‚2217§Ãe¯Ò'ÿÌO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚211411836536832 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54660032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚207623765810176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13812098 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚211411941598208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3898650177536 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784504063552373e+09 _step‚435 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚211411941598208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54649889 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95612084 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40962197 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚204929366231824 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚202221549356032 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚198322924082176 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54660032 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚204929349192464 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚208705673071066 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3788070726656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23732664 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚207623782849536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40962197 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3898836882432 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23732667 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3776220690768 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚207623765810176 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3788158748672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37545013 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚198322899178496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚208705569883232 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13812346 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3776306839242 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3788158748672 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40962195 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚207623782849536 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚202221760964608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚208705673071066 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3788070726656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23732664 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚207623782849536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚40962197 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3898836882432 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23732667 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3776220690768 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚207623765810176 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3788158748672 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37545013 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚198322899178496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚208705569883232 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13812346 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3776306839242 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3788158748672 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚40962195 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚207623782849536 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚202221760964608 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚211411836536832 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54660032 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚207623765810176 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13812098 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚211411941598208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3898650177536 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 $ _timestamp‚1.784504063552373e+09 _step‚435 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚211411941598208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54649889 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚95612084 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚40962197 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚204929366231824 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚202221549356032 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚198322924082176 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54660032 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚204929349192464 IDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95639462 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚204966221260224 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3788836865536 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23736957 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚208743117902436 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚207661149216768 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40969518 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚198358619239424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37551487 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40969518 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚207661149216768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚207661132177408 _runtime‚ 2217.88304883 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3788836865536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3776982790798 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚204966238299584 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7845040638197825e+09 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚211449881013760 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54669944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95639462 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40969516 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 BDevi3žÈD6ceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23736954 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚207661132177408 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3788748836352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40969516 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚202258150588928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95639462 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40969516 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23736954 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚207661132177408 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3788748836352 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40969516 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚202258150588928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95639462 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚204966221260224 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3788836865536 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23736957 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚208743117902436 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚207661149216768 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚40969518 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚198358619239424 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37551487 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40969518 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚207661149216768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚207661132177408 _runtime‚ 2217.88304883 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3788836865536 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3776982790798 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚204966238299584 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7845040638197825e+09 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚211449881013760 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54669944´æõÑ2„- _wandbruntime‚2217  _runtime‚2217ˆ?mi… 0loss/FRt1/assignment_loss‚2.5921271400156813 2loss/FBt2/assignment_loss‚0.029732047009242906 /loss/FRt2/detection_loss‚1.5753260813100989 &loss/total_loss‚10.507935704206508 $ _timestamp‚1.784504063820797e+09 0loss/FBt1/assignment_loss‚0.2478692988355135 1loss/SRqqt1/detection_loss‚1.2214914232936522 /loss/FBt2/detection_loss‚0.4908200577867053 trainer/global_step‚5399 _runtime‚2217.883623331 _step‚437 0loss/FRt2/assignment_loss‚1.0319004877138946 2loss/SRqqt1/assignment_loss‚0.7501814047334135 /loss/FRt1/detection_loss‚0.6295440898744823 1loss/SRqqt2/detection_loss‚0.6844016263744391 2loss/SRqqt2/assignment_loss‚0.7046513472767411 /loss/FBt1/detection_loss‚0.5498906999826432 epoch‚6ùƒ$ †† 2loss/FBt2/assignment_loss‚0.029732047009242906 /loss/FRt2/detection_loss‚1.5753260813100989 &loss/total_loss‚10.507935704206508 $ _timestamp‚1.784504063820797e+09 0loss/FBt1/assignment_loss‚0.2478692988355135 1loss/SRqqt1/detection_loss‚1.2214914232936522 /loss/FBt2/detection_loss‚0.4908200577867053 trainer/global_step‚5399 _runtime‚2217.883623331 _step‚437 0loss/FRt2/assignment_loss‚1.0319004877138946 2loss/SRqqt1/assignment_loss‚0.7501814047334135 /loss/FRt1/detection_loss‚0.6295440898744823 1loss/SRqqt2/detection_loss‚0.6844016263744391 2loss/SRqqt2/assignment_loss‚0.7046513472767411 /loss/FBt1/detection_loss‚0.5498906999826432 epoch‚6 0loss/FRt1/assignment_loss‚2.5921271400156813µQ01Pi‡‚(:Þ ¾õҀщÀ disk.dm-1.out‚1342.775390625( proc.memory.availableMB‚ 129.99609375) proc.memory.percent‚17.49107837677002 disk.nvme0n1.in‚91.03125 disk.nvme1n1.in‚ 10.48828125 disk.dm-0.in‚ 101.51953125# proc.memory.rssMB‚ 2865.73828125 cpu‚26.58031088942881 proc.cpu.threads‚17 disk.md0.out‚ 65.91796875 network.sent‚ 7.063814e+06 network.recv‚651549$ disk./.usageGB‚1164.410530090332 disk.nvme0c0n1.in‚91.03125( disk.nvme0c0n1.out‚13970.23974609375' disk.nvme1c1n1.out‚3027.01513671875% disk.nvme1n1.out‚3027.01513671875) disk./.usagePercent‚75.84501091615905& disk.nvme0n1.out‚13970.23974609375! disk.nvme1c1n1.in‚ 10.48828125 disk.md0.in‚0 disk.dm-0.out‚ 15513.953125 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚99.20656681060791 disk.dm-1.in‚0v( œˆ‚(:‘ ¾õÒ‚ï±À gpu.0.gpu‚90.0 gpu.0.memory‚12+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚198.225' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.07499999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 \Qg‰‚(:Ü ¾õÒ ËÖ¿ cpu‚26.5756723168499# disk.dm-0.out‚15516.61279296875) proc.memory.percent‚17.49107837677002( disk.nvme0c0n1.out‚13973.06201171875! disk.nvme1c1n1.in‚ 10.48828125% disk.nvme1c1n1.out‚3027.630859375 disk.nvme1n1.in‚ 10.48828125 disk.md0.in‚0 disk.md0.out‚ 66.234375# proc.memory.rssMB‚ 2865.73828125 proc.cpu.threads‚17 network.sent‚ 7.115263e+06 network.recv‚655914) disk./.usagePercent‚75.84504073304048$ disk./.usageGB‚1164.410987854004 disk.nvme0c0n1.in‚91.03125# disk.nvme1n1.out‚3027.630859375 disk.dm-1.out‚1342.869140625( proc.memory.availableMB‚ 129.99609375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚91.03125& disk.nvme0n1.out‚13973.06201171875 disk.dm-0.in‚ 101.51953125 disk.dm-1.in‚0$ memory_percent‚99.20656681060791ø"g›Š‚(: ¾õҘ܄À gpu.0.gpu‚50.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚198.486' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.16199999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,»ê®2‹- _wandbruntime‚2236  _runtime‚2236¼DȌ trainer/global_step‚5449 $ _timestamp‚1.784504082863086e+09 _runtime‚2236.926257116 _step‚438 ' lr-AdamW/pg1‚0.00010829162247002253 ' lr-AdamW/pg2‚0.00010829162247002253F›ÍÌÍÇ ' lr-AdamW/pg1‚0.00010829162247002253 ' lr-AdamW/pg2‚0.00010829162247002253 trainer/global_step‚5449 $ _timestamp‚1.784504082863086e+09 _runtime‚2236.926257116 _step‚438¶°"Aq2Ž- _wandbruntime‚2236  _runtime‚2236öulÔ'ÎO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41328247 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚209492084171776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41328245 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚209492101211136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3822008468480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204041485388288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96474886 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37882890 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3933620933120 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55146641 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚200107657040896 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213314197701632 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚213314197701632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3822008468480 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55146641 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚439 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚209492084171776 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚210583002746640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23947167 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚210583105934474 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 % _timestamp‚1.7845040828639736e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55156784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41328245 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚200107681944576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213314092640256 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚209492101211136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3822096490496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204041277974016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213314092640256 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚206772952580464 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41328247 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96485031 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚206772969619824 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3810136314650 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 _runtime‚2236.930439414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3933803443712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13935966 trainer/global_step‚5449 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55156784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3810050166176÷µ…2fÓO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚23947164 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213314092640256 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚209492101211136 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3822096490496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204041277974016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213314092640256 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚206772952580464 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚41328247 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚96485031 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚206772969619824 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3810136314650 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 _runtime‚2236.930439414 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3933803443712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13935966 trainer/global_step‚5449 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55156784 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚240 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3810050166176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚41328247 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚209492084171776 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41328245 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚209492101211136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3822008468480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204041485388288 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96474886 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37882890 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3933620933120 L?DeviceStatsMonitor.on_train_batch_start/active.small_p\vàs ool.freed‚55146641 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚200107657040896 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213314197701632 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚213314197701632 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3822008468480 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55146641 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚439 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚209492084171776 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚210583002746640 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23947167 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚210583105934474 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 % _timestamp‚1.7845040828639736e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55156784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41328245 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚200107681944576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96492105 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚209529450539008 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚213352137117184 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3934501287936 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55166696 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 trainer/global_step‚5449 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37889644 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204077878389760 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3822686578176 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚200143352198144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚206809841687584 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 _runtime‚ 2237.50651273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚209529450539008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13938190 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55156539 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚209529467578368 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 _step‚440 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213352242185728 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚213352242185728 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96492105 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚206809824648224 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96502264 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 % _timestamp‚1.7845040834432054e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3822774607360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13938442 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210620550765844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55166696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37889899 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41335566 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96502264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41335568 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204077673079808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚209529467578368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚200143377101824 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255A¯¼Qâ&“ÜM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37889644 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204077878389760 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3822686578176 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚200143352198144 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚206809841687584 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 _runtime‚ 2237.50651273 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚209529450539008 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13938190 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55156539 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚209529467578368 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 _step‚440 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213352242185728 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚213352242185728 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚96492105 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚206809824648224 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96502264 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚252 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 % _timestamp‚1.7845040834432054e+09 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3822774607360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13938442 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210620550765844 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55166696 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37889899 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41335566 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96502264 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41335568 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204077673079808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚209529467578368 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚200143377101824 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚255 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23951454 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚210620653953790 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96492105 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚209529450539008 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚213352137117184 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3934501287936 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55166696 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 trainer/global_step‚5449¸dÀ¸Ÿ2”- _wandbruntime‚2237  _runtime‚2237O‚øí•ÿ 2loss/SRqqt1/assignment_loss‚0.6708604918730159 1loss/SRqqt1/detection_loss‚1.3085229409715304 % _timestamp‚1.7845040834445894e+09 _step‚441 0loss/FRt2/assignment_loss‚0.9721073792739348 2loss/SRqqt2/assignment_loss‚0.6361590111609842 /loss/FRt1/detection_loss‚0.5963873708221215 /loss/FBt2/detection_loss‚0.7074930891864032 %loss/total_loss‚12.08184715106513 epoch‚6 0loss/FBt1/assignment_loss‚0.5249083484364936 /loss/FRt2/detection_loss‚1.4244986053690258 1loss/SRqqt2/detection_loss‚0.6186772797388401 .loss/FBt1/detection_loss‚1.917332302822615 trainer/global_step‚5449 _runtime‚2237.510592228 0loss/FRt1/assignment_loss‚2.5911966217802718 1loss/FBt2/assignment_loss‚0.11370370962989322è”×Ü –„ 0loss/FRt2/assignment_loss‚0.9721073792739348 2loss/SRqqt2/assignment_loss‚0.6361590111609842 /loss/FRt1/detection_loss‚0.5963873708221215 /loss/FBt2/detection_loss‚0.7074930891864032 %loss/total_loss‚12.08184715106513 epoch‚6 0loss/FBt1/assignment_loss‚0.5249083484364936 /loss/FRt2/detection_loss‚1.4244986053690258 1loss/SRqqt2/detection_loss‚0.6186772797388401 .loss/FBt1/detection_loss‚1.917332302822615 trainer/global_step‚5449 _runtime‚2237.510592228 0loss/FRt1/assignment_loss‚2.5911966217802718 1loss/FBt2/assignment_loss‚0.11370370962989322 2loss/SRqqt1/assignment_loss‚0.6708604918730159 1loss/SRqqt1/detection_loss‚1.3085229409715304 % _timestamp‚1.7845040834445894e+09 _step‚441¹üO0Nl—‚(:á Ÿ¾õÒò¿›À' disk.nvme1c1n1.out‚3028.16748046875% disk.nvme1n1.out‚3028.16748046875 disk.md0.out‚ 66.37109375 disk.dm-0.in‚ 101.51953125 disk.dm-0.out‚ 15525.0859375 disk.dm-1.in‚0# proc.memory.rssMB‚ 2865.73828125 network.sent‚ 7.131071e+06) disk./.usagePercent‚75.84506408959759% disk./.usageGB‚1164.4113464355469% disk.nvme0n1.out‚13981.4365234375 disk.nvme0c0n1.in‚91.03125' disk.nvme0c0n1.out‚13981.4365234375$ memory_percent‚99.20675754547119( proc.memory.availableMB‚ 129.96484375 network.recv‚659217 disk.nvme0n1.in‚91.03125 disk.nvme1n1.in‚ 10.48828125 disk.rbd0.in‚0) proc.memory.percent‚17.49107837677002 cpu‚26.571403350446182 proc.cpu.threads‚17! disk.nvme1c1n1.in‚ 10.48828125 disk.md0.in‚0" disk.dm-1.out‚1342.99658203125 disk.rbd0.out‚0ÍbqÍ›˜‚(: Ÿ¾õÒ³¿ÄÀ gpu.0.gpu‚57.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚202.647' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.54899999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vóS2™- _wandbruntime‚2256  _runtime‚2256 ˜Lɚà _step‚442 ' lr-AdamW/pg1‚0.00010820964132366024 ' lr-AdamW/pg2‚0.00010820964132366024 trainer/global_step‚5499 % _timestamp‚1.7845041024304726e+09 _runtime‚2256.493597829¤xPÎ›È ' lr-AdamW/pg1‚0.00010820964132366024 ' lr-AdamW/pg2‚0.00010820964132366024 trainer/global_step‚5499 % _timestamp‚1.7845041024304726e+09 _runtime‚2256.493597829 _step‚442ºêXŒ2œ- _wandbruntime‚2256  _runtime‚2256ööl(÷ÎO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55653536 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3856034232320 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41694295 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚205860990015488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚211360402533376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201892439806976 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚208616555968464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚215216348743680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55643393 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3856034232320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55653536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 % _timestamp‚1.7845041024314117e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14060336 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚215216453805056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚205861203721216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚212460538797882 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚208616573007824 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97337688 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚215216453805056 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3968575112192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _runtime‚2256.494516651 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚215216348743680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24161664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚212460435610048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚211360402533376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14060084 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38222003 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3843879641584 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚211360419572736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104µaÙ÷Ù'žÓO BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 % _timestamp‚1.7845041024314117e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14060336 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚215216453805056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚205861203721216 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚212460538797882 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚208616573007824 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚97337688 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚215216453805056 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3968575112192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _runtime‚2256.494516651 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚215216348743680 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24161664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚212460435610048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚211360402533376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14060084 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38222003 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3843879641584 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚211360419572736 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55653536 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3856034232320 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41694295 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚205860990015488 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚211360402533376 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚201892439806976 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚208616555968464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚215216348743680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55643393 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3856034232320 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55653536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816»žsÁÆ2Ÿ- _wandbruntime‚2256  _runtime‚2256©¹¥ŸÞ& ØM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _step‚444 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55663448 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚211397768900608 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41701616 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41701616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚215254393220608 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55653291 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55653291 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97365066 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41701618 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚205897588641280 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38228779 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚212497983629252 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41701618 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚2256.761148007 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3856712349184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97354907 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38228539 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚211397768900608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14062585 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3856624320000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14062822 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24165957 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚201928110060544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3969453677056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97365066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3844555593028 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97354907 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚205897383331328 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚212498086817198 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0eQ¨  ¡ÝM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55663448 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚211397768900608 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41701616 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41701616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚215254393220608 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚55653291 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚55653291 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚97365066 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41701618 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚205897588641280 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38228779 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚212497983629252 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41701618 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚2256.761148007 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3856712349184 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚97354907 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38228539 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚211397768900608 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14062585 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3856624320000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14062822 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24165957 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚201928110060544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3969453677056 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚97365066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3844555593028 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚97354907 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚205897383331328 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚212498086817198 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _step‚444 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/rÅM0Ìeserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482¼Úå<Ê2¢- _wandbruntime‚2256  _runtime‚2256D F£€ 2loss/FBt2/assignment_loss‚0.030401569924876526 /loss/FBt1/detection_loss‚1.2028529791213158 &loss/total_loss‚11.085032190805341 _step‚445 2loss/SRqqt2/assignment_loss‚0.6691010960810061 0loss/FBt1/assignment_loss‚0.1799292712696359 /loss/FRt1/detection_loss‚0.5306730812959498 0loss/SRqqt1/detection_loss‚1.096127504529744 epoch‚6 _runtime‚2256.761899339 0loss/FRt1/assignment_loss‚2.6744219119477477 0loss/FRt2/assignment_loss‚0.9776978012451456 2loss/SRqqt1/assignment_loss‚0.8478747441889656 .loss/FRt2/detection_loss‚1.606682474434391 /loss/FBt2/detection_loss‚0.5933267392000534 1loss/SRqqt2/detection_loss‚0.6759430175665099 trainer/global_step‚5499 % _timestamp‚1.7845041026989036e+09¾,˜ ¤… 2loss/SRqqt2/assignment_loss‚0.6691010960810061 0loss/FBt1/assignment_loss‚0.1799292712696359 /loss/FRt1/detection_loss‚0.5306730812959498 0loss/SRqqt1/detection_loss‚1.096127504529744 epoch‚6 _runtime‚2256.761899339 0loss/FRt1/assignment_loss‚2.6744219119477477 0loss/FRt2/assignment_loss‚0.9776978012451456 2loss/SRqqt1/assignment_loss‚0.8478747441889656 .loss/FRt2/detection_loss‚1.606682474434391 /loss/FBt2/detection_loss‚0.5933267392000534 1loss/SRqqt2/detection_loss‚0.6759430175665099 trainer/global_step‚5499 % _timestamp‚1.7845041026989036e+09 2loss/FBt2/assignment_loss‚0.030401569924876526 /loss/FBt1/detection_loss‚1.2028529791213158 &loss/total_loss‚11.085032190805341 _step‚445½A›·›¥‚(: ®¾õÒªäœÀ gpu.0.gpu‚59.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚201.802' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.26733333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[pkðl¦‚(:á ®¾õÒ–ìÓÁ proc.cpu.threads‚17) disk./.usagePercent‚75.82173983411319" disk.dm-1.out‚1343.09814453125 disk.nvme0n1.in‚91.03125& disk.nvme0n1.out‚14003.87841796875% disk.nvme1n1.out‚3029.07373046875# disk.dm-0.out‚15546.86376953125) proc.memory.percent‚17.49107837677002 network.sent‚ 7.19353e+06% disk./.usageGB‚1164.0532608032227 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚14003.87841796875 disk.md0.in‚0 disk.dm-0.in‚ 101.51953125 network.recv‚663599 disk.nvme0c0n1.in‚91.03125! disk.nvme1c1n1.in‚ 10.48828125' disk.nvme1c1n1.out‚3029.07373046875 disk.md0.out‚67.0625( proc.memory.availableMB‚ 129.89453125# proc.memory.rssMB‚ 2865.73828125 cpu‚26.56694927528717 disk.nvme1n1.in‚ 10.48828125$ memory_percent‚99.20718669891357¹eÃ2§- _wandbruntime‚2276  _runtime‚22760Q½É¨Ã ' lr-AdamW/pg1‚0.00010812682495113178 ' lr-AdamW/pg2‚0.00010812682495113178 trainer/global_step‚5549 % _timestamp‚1.7845041220050714e+09 _runtime‚2276.069565921 _step‚446¢:ôªÎ©È ' lr-AdamW/pg1‚0.00010812682495113178 ' lr-AdamW/pg2‚0.00010812682495113178 trainer/global_step‚5549 % _timestamp‚1.7845041220050714e+09 _runtime‚2276.069565921 _step‚446¾;ë)½2ª- _wandbruntime‚2276  _runtime‚2276K7PÓ'«ÍO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚214337868473456 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98200490 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚447 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24376164 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3889971974144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚214337971661290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3889883952128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192996352 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56150288 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 $ _timestamp‚1.784504122005761e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42060345 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42060345 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚207680906606080 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚203677197669376 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56140145 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚217118709908480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚213228720894976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98210635 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4003708936704 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56150288 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56140145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217900032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚5549 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3889971974144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3889883952128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚217118604847104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚217118604847104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38559083 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14182919 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42060347 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38559345 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚203677172765696 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚2276.069651001 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚210460176395824 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3877795265466 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14183178 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚207680688706048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚213228737934336 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚213228720894976 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 •:ÕØ'¬ÒO QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42060347 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38559345 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚203677172765696 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚2276.069651001 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚210460176395824 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3877795265466 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14183178 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚207680688706048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚213228737934336 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚213228720894976 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚214337868473456 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98200490 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚447 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24376164 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3889971974144 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚214337971661290 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3889883952128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192996352 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56150288 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 $ _timestamp‚1.784504122005761e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42060345 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42060345 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚207680906606080 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚203677197669376 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56140145 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚217118709908480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚213228720894976 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98210635 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4003708936704 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56150288 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56140145 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217900032 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚5549 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3889971974144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3889883952128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚217118604847104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚217118604847104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38559083 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14182919 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0¿À*C§2­- _wandbruntime‚2276  _runtime‚2276Õö~°°®×M B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42067666 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42067668 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3878471217022 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚213266087262208 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56150043 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24380454 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4004402168832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚207717087588352 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚213266104301568 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56160200 ZFDeviceStatsMonitorÍÞ$- .on_train_batch_end/allocated_bytes.large_pool.freed‚213266087262208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚203712892826624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42067666 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚217156754392576 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚214375416492660 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4004219665408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.784504122273842e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _step‚448 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚217156649324032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚213266104301568 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98217709 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚217156754392576 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚210497031424224 _runtime‚2276.337108956 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38565861 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚203712867922944 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98217709 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98227868 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56160200 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚214375519680606 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3890562061824 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56160200 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚213266087262208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚203712892826624 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚42067666 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚217156754392576 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚214375416492660 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4004219665408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.784504122273842e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _step‚448 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚217156649324032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚213266104301568 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98217709 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚217156754392576 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚210497031424224 _runtime‚2276.337108956 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38565861 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚203712867922944 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98217709 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98227868 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56160200 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚214375519680606 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3890562061824 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3878471217022 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚213266087262208 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56150043 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24380454 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4004402168832 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚207717087588352 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0À?°A2°- _wandbruntime‚2276  _runtime‚2276Ýæù˱þ epoch‚6 trainer/global_step‚5549 0loss/FRt1/assignment_loss‚2.5694969142242132 /loss/FRt2/assignment_loss‚1.089482843875885 /loss/FRt2/detection_loss‚1.4607724916729097 /loss/SRqqt1/detection_loss‚1.41215081375215 1loss/SRqqt2/detection_loss‚0.6895156295013888 _runtime‚2276.337688257 &loss/total_loss‚11.106660116559155 0loss/FBt2/assignment_loss‚0.0538687145042556 /loss/FBt1/detection_loss‚1.0197758922981546 % _timestamp‚1.7845041222748587e+09 _step‚449 2loss/SRqqt1/assignment_loss‚0.9910338742896742 2loss/SRqqt2/assignment_loss‚0.5818641914278601 1loss/FBt1/assignment_loss‚0.09626686412813494 /loss/FRt1/detection_loss‚0.5942658021115245 /loss/FBt2/detection_loss‚0.5481660847730047á_øz ²ƒ 2loss/SRqqt1/assignment_loss‚0.9910338742896742 2loss/SRqqt2/assignment_loss‚0.5818641914278601 1loss/FBt1/assignment_loss‚0.09626686412813494 /loss/FRt1/detection_loss‚0.5942658021115245 /loss/FBt2/detection_loss‚0.5481660847730047 epoch‚6 trainer/global_step‚5549 0loss/FRt1/assignment_loss‚2.5694969142242132 /loss/FRt2/assignment_loss‚1.089482843875885 /loss/FRt2/detection_loss‚1.4607724916729097 /loss/SRqqt1/detection_loss‚1.41215081375215 1loss/SRqqt2/detection_loss‚0.6895156295013888 _runtime‚2276.337688257 &loss/total_loss‚11.106660116559155 0loss/FBt2/assignment_loss‚0.0538687145042556 /loss/FBt1/detection_loss‚1.0197758922981546 % _timestamp‚1.7845041222748587e+09 _step‚449Áá…‰”g³‚(:Ü ½¾õÒîý™À network.recv‚667469! disk.dm-1.out‚1362.2568359375 disk.rbd0.out‚0& disk.nvme1c1n1.out‚3048.9501953125$ memory_percent‚99.20775890350342( proc.memory.availableMB‚ 129.80078125 network.sent‚ 7.25588e+06) disk./.usagePercent‚75.82174157343127 disk.md0.in‚0 disk.dm-0.in‚ 101.51953125 disk.dm-0.out‚15550.16796875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚91.03125& disk.nvme0c0n1.out‚14007.072265625% disk./.usageGB‚1164.0532875061035 disk.nvme0n1.in‚91.03125$ disk.nvme0n1.out‚14007.072265625! disk.nvme1c1n1.in‚ 10.48828125 disk.nvme1n1.in‚ 10.48828125# proc.memory.rssMB‚ 2865.73828125) proc.memory.percent‚17.49107837677002 disk.md0.out‚ 67.33984375 disk.dm-1.in‚0$ disk.nvme1n1.out‚3048.9501953125 cpu‚26.562950499504787 proc.cpu.threads‚17BŸ›´‚(: ½¾õÒ¢àÁÀ gpu.0.gpu‚63.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚190.613' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚63.53766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TÚãmµ‚(:â ̾õÒÆ’Ä¿) proc.memory.percent‚17.49107837677002) disk./.usagePercent‚75.80624052220729% disk./.usageGB‚1163.8153076171875 disk.nvme0n1.in‚91.03125& disk.nvme0n1.out‚14011.04443359375 disk.nvme1n1.in‚ 10.48828125 disk.dm-1.in‚0! disk.dm-1.out‚1572.1396484375 disk.rbd0.out‚0 network.recv‚672010 disk.nvme0c0n1.in‚91.03125( disk.nvme0c0n1.out‚14011.04443359375' disk.nvme1c1n1.out‚3259.32958984375 disk.dm-0.in‚ 101.51953125& proc.memory.availableMB‚ 130.046875 cpu‚26.55855177570623! disk.nvme1c1n1.in‚ 10.48828125 disk.md0.out‚ 67.74609375# proc.memory.rssMB‚ 2865.73828125 proc.cpu.threads‚17 network.sent‚ 7.305478e+06% disk.nvme1n1.out‚3259.32958984375 disk.md0.in‚0" disk.dm-0.out‚15553.7529296875 disk.rbd0.in‚0$ memory_percent‚99.20625686645508 ÐÇœ¶‚(:‘ ̾õÒ«ÚÀ gpu.0.gpu‚83.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚196.543' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.51433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Šþ6 2·- _wandbruntime‚2295  _runtime‚2295e~ÄÉɸà ' lr-AdamW/pg2‚0.00010804317467890945 trainer/global_step‚5599 % _timestamp‚1.7845041415611572e+09 _runtime‚2295.625581656 _step‚450 ' lr-AdamW/pg1‚0.00010804317467890945Ìs•„Î¹È ' lr-AdamW/pg1‚0.00010804317467890945 ' lr-AdamW/pg2‚0.00010804317467890945 trainer/global_step‚5599 % _timestamp‚1.7845041415611572e+09 _runtime‚2295.625581656 _step‚450 z^2º- _wandbruntime‚2295  _runtime‚2295Mé¿Ô'»ÎO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209500630340096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209500420828672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24590667 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚215097056295936 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4038490200576 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚219020966011904 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚215097056295936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚219020860950528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205461955531776 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚215097039256576 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 trainer/global_step‚5599 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚212303762744464 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99073437 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14306346 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56647040 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42426397 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7845041415618634e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38897010 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚212303779783824 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38897262 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14306595 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24590664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42426395 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42426395 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚219020860950528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56636897 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56636897 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99063292 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚219020966011904 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚216215301336864 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚451 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚215097039256576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3923909715968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42426397 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚2295.625648266 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4038674808320 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3911624740874 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3923821693952 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3911538592400 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205461930628096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3923909715968 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚216215404524698 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3923821693952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238DD’ao¼ÓO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4038490200576 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚219020966011904 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚215097056295936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚219020860950528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205461955531776 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚215097039256576 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 trainer/global_step‚5599 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚212303762744464 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99073437 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14306346 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚56647040 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42426397 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7845041415618634e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38897010 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚212303779783824 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38897262 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14306595 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24590664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42426395 OEj>i¼jDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚42426395 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚219020860950528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚56636897 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚56636897 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99063292 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚219020966011904 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚216215301336864 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚451 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚215097039256576 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3923909715968 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42426397 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚2295.625648266 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4038674808320 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3911624740874 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3923821693952 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3911538592400 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205461930628096 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3923909715968 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚216215404524698 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3923821693952 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209500630340096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚209500420828672 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24590667 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚215097056295936 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215134405623808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209537019450368 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3924499803648 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚215134422663168 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4039368761344 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42433716 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99090670 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14309061 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 _runtime‚2295.891017925 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3912300692430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3912214543844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24594957 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚219058905427456 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99090670 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14308814 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24594954 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚215134422663168 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56656952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4039190452224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42433718 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205497650689024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚216252849356068 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99080511 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38903768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56656952 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚219059010496000 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99080511 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚209536816237568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712bÃäçâ&¿ÜM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38903768 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56656952 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚219059010496000 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99080511 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚209536816237568 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56646795 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56646795 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38904018 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚216252952544014 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215134405623808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209537019450368 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3924499803648 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚215134422663168 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4039368761344 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42433716 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99090670 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14309061 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 _runtime‚2295.891017925 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3912300692430 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3912214543844 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24594957 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚219058905427456 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99090670 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14308814 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24594954 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚215134422663168 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56656952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4039190452224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42433718 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205497650689024 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚216252849356068 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99080511 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0Ä€&G2À- _wandbruntime‚2295  _runtime‚2295ËAL@Áÿ 1loss/SRqqt2/detection_loss‚0.6346664875620559 /loss/FBt1/detection_loss‚0.8740813584677778 _runtime‚2295.891606336 _step‚453 1loss/FBt2/assignment_loss‚0.07943464729235937 /loss/FBt2/detection_loss‚0.4929854741766773 trainer/global_step‚5599 /loss/FRt1/assignment_loss‚2.649911731806196 1loss/SRqqt1/assignment_loss‚0.669212051055261 /loss/FRt1/detection_loss‚0.5499229576710808 .loss/FRt2/detection_loss‚1.284724507253361 1loss/SRqqt1/detection_loss‚1.2346600296208634 1loss/FBt1/assignment_loss‚0.19434131594091644 &loss/total_loss‚10.342991180256117 epoch‚6 % _timestamp‚1.7845041418287935e+09 0loss/FRt2/assignment_loss‚1.0197950685752257 2loss/SRqqt2/assignment_loss‚0.6592555508343435}­ª˜ „ 1loss/FBt1/assignment_loss‚0.19434131594091644 &loss/total_loss‚10.342991180256117 epoch‚6 % _timestamp‚1.7845041418287935e+09 0loss/FRt2/assignment_loss‚1.0197950685752257 2loss/SRqqt2/assignment_loss‚0.6592555508343435 1loss/SRqqt2/detection_loss‚0.6346664875620559 /loss/FBt1/detection_loss‚0.8740813584677778 _runtime‚2295.891606336 _step‚453 1loss/FBt2/assignment_loss‚0.07943464729235937 /loss/FBt2/detection_loss‚0.4929854741766773 trainer/global_step‚5599 /loss/FRt1/assignment_loss‚2.649911731806196 1loss/SRqqt1/assignment_loss‚0.669212051055261 /loss/FRt1/detection_loss‚0.5499229576710808 .loss/FRt2/detection_loss‚1.284724507253361 1loss/SRqqt1/detection_loss‚1.2346600296208634ÅXýi‹VÂ(:Ë Û¾õÒÖõß¿ disk.nvme1n1.in‚ 10.48828125 disk.md0.out‚ 67.890625 cpu‚26.569893803917495 disk.dm-0.in‚ 101.51953125 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme1c1n1.in‚ 10.48828125& disk.nvme1c1n1.out‚3383.2080078125$ disk.nvme1n1.out‚3383.2080078125% disk./.usageGB‚1163.8317489624023 disk.dm-1.in‚0 disk.dm-1.out‚1694.064453125# memory_percent‚99.2218017578125! proc.memory.availableMB‚127.5 proc.cpu.threads‚17 network.sent‚ 7.324235e+06' disk./.usagePercent‚75.807311445198" disk.dm-0.out‚15574.5595703125 disk.nvme0c0n1.in‚91.03125! disk.nvme0n1.out‚ 14030.234375 disk.md0.in‚0" proc.memory.rssMB‚ 2868.0078125) proc.memory.percent‚17.50493049621582 network.recv‚675223# disk.nvme0c0n1.out‚ 14030.234375 disk.nvme0n1.in‚91.03125Þ¨'›Ä‚(: Û¾õÒ²ŠÀ gpu.0.gpu‚38.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚198.923' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.30766666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E9‹q2Å- _wandbruntime‚2314  _runtime‚2314<!³ôÆöz jet/accuracy_2_of_5‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37393122911453247 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_4‚0.7936181426048279 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9980935454368591 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03679138422012329 *jet/accuracy_1_of_3‚0.9137717485427856 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.625365138053894 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7596637606620789 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37288087606430054 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4628288745880127 . Purity/2FRt*SRqqt1FBt/FRt_purity‚ 0.4921875 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9969596862792969 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925780296325684 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.49018582701683044 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6795243620872498 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17688894271850586 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11487959325313568 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15667235851287842 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.05000000074505806 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 _runtime‚2314.187795973 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9253242015838623 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +jet/accuracy_3_of_4‚0.09063562750816345 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9914351105690002 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6074087619781494 trainer/global_step‚5606 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9951246976852417 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21168695390224457 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6223843097686768 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0357142873108387 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6869350075721741 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17364563047885895 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7954755425453186 particle/accuracy_1_of_2‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.13675253093242645 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24378910660743713 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03679138422012329 particle/accuracy_3_of_4‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2Fª²ßЈ+Rt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0357142873108387 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1598207652568817 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3718893527984619 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.841666579246521 _step‚454 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996954619884491 particle/accuracy_1_of_1‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.654104471206665 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7275370955467224 jet/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6344208717346191 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12083333730697632 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987784028053284 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6835681796073914 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.019949812442064285 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8441795110702515 *jet/accuracy_1_of_1‚0.2986662983894348 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1367419958114624 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5251232981681824 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5328465700149536 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08220867067575455 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.050803154706954956 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9821428656578064 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8361549377441406 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03393911197781563 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 particle/accuracy_2_of_3‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24086788296699524 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11497653275728226 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 5 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.99627685546875 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1409471184015274 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02372237667441368 )jet/accuracy_1_of_4‚0.967913031578064 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9971655011177063 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9844576716423035 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5213386416435242 particle/accuracy_2_of_2‚1 .particle/accuracy_4_of_4‚0.954865574836731 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.27843087911605835 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9352306127548218 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6096990704536438 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5880162715911865 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.394514799118042 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_3‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5208333730697632 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012341849505901337 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9922288060188293 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.38391390442848206 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 *jet/accuracy_2_of_3‚0.5287247896194458 +particle/specificity‚0.9050315022468567 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.14275109767913818 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14766010642051697 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4469679892063141 particle/accuracy_1_of_4‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8065788745880127 epoch‚6 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.048700038343667984 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7223385572433472 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7692259550094604 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9962725043296814 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09804118424654007 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6344208717346191 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.39393937587738037 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12831370532512665 particle/accuracy_2_of_4‚1 particle/accuracy_1_of_5‚1 particle/accuracy_4_of_5‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4932778775691986 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09852510690689087 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.05185007303953171 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8781686425209045 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6238932013511658 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 'particle/f_score‚0.6395764350891113 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14766010642051697 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7655442357063293 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8712139129638672 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6784118413925171 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15671995282173157 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2777777910232544 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7552547454833984 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9750000238418579 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6874231100082397 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4068189263343811 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2183353155851364 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4654218256473541 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04814106971025467 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4804416000843048 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7418217658996582 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4919041097164154 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6308664679527283 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7071727514266968 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6133804321289062 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12958407402038574 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9954544901847839 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.15995649993419647 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7993426322937012 +particle/sensitivity‚0.6334576606750488 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8503759503364563 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.990462064743042 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9974288940429688 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *loss/val_total_loss‚10.419010042664329 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.042845312505960464 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.05000000074505806 particle/accuracy_2_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14236323535442352 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.699999988079071 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5370702147483826 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12534311413764954 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 $ _timestamp‚1.784504160124204e+09 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5339846611022949 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6892868876457214 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.748446524143219 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19432727992534637 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8441795110702515 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5955602526664734 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,jet/accuracy_4_of_4‚0.014682539738714695 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8230767846107483 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.699999988079071 6validation_average_jet_accuracy‚0.3452565670013428 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7346153259277344 jet/accuracy_3_of_5‚1 +jet/accuracy_3_of_3‚0.06594487279653549 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.995820164680481 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3611110746860504 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7477471232414246 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 (particle/accuracy‚0.8467183709144592 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15959374606609344 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1395649015903473 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14688479900360107 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.037718288600444794 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4749999940395355 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8198106288909912 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534² ì=Çûz 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6795243620872498 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.17688894271850586 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11487959325313568 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.15667235851287842 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.05000000074505806 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 _runtime‚2314.187795973 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9253242015838623 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 +jet/accuracy_3_of_4‚0.09063562750816345 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9914351105690002 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6074087619781494 trainer/global_step‚5606 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9951246976852417 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21168695390224457 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6223843097686768 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0357142873108387 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6869350075721741 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.17364563047885895 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7954755425453186 particle/accuracy_1_of_2‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.13675253093242645 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24378910660743713 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03679138422012329 particle/accuracy_3_of_4‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0357142873108387 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1598207652568817 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3718893527984619 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.841666579246521 _step‚454 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.996954619884491 particle/accuracy_1_of_1‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 8"Purity/0FRt*SRqqt*FBt/event_purity‚0.654104471206665 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7275370955467224 jet/accuracy_1_of_5‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6344208717346191 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12083333730697632 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9987784028053284 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6835681796073914 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.019949812442064285 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8441795110702515 *jet/accuracy_1_of_1‚0.2986662983894348 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1367419958114624 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5251232981681824 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5328465700149536 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.08220867067575455 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.050803154706954956 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9821428656578064 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8361549377441406 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03393911197781563 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 particle/accuracy_2_of_3‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24086788296699524 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11497653275728226 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 5 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.99627685546875 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1409471184015274 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.02372237667441368 )jet/accuracy_1_of_4‚0.967913031578064 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9971655011177063 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9844576716423035 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5213386416435242 particle/accuracy_2_of_2‚1 .particle/accuracy_4_of_4‚0.954865574836731 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.27843087911605835 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9352306127548218 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6096990704536438 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5880162715911865 8"Purity/0FRt2SRqqt0FBt/event_purity‚0.394514799118042 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 particle/accuracy_3_of_3‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5208333730697632 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.012341849505901337 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9922288060188293 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.38391390442848206 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 *jet/accuracy_2_of_3‚0.5287247896194458 +particle/specificity‚0.9050315022468567 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.14275109767913818 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14766010642051697 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4469679892063141 particle/accuracy_1_of_4‚1 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8065788745880127 epoch‚6 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.048700038343667984 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7223385572433472 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 *jet/accuracy_1_of_2‚0.7692259550094604 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9962725043296814 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09804118424654007 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6344208717346191 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.39393937587738037 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12831370532512665 particle/accuracy_2_of_4‚1 particle/accuracy_1_of_5‚1 particle/accuracy_4_of_5‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4932778775691986 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09852510690689087 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.05185007303953171 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8781686425209045 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6238932013511658 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 'particle/f_score‚0.6395764350891113 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14766010642051697 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7655442357063293 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8712139129638672 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6784118413925171 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15671995282173157 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2777777910232544 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7552547454833984 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9750000238418579 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 particle/accuracy_3_of_5‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6874231100082397 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4068189263343811 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2183353155851364 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4654218256473541 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.04814106971025467 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4804416000843048 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7418217658996582 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.4919041097164154 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6308664679527283 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7071727514266968 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6133804321289062 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12958407402038574 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9954544901847839 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.15995649993419647 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7993426322937012 +particle/sensitivity‚0.6334576606750488 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8503759503364563 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.990462064743042 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9974288940429688 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 *loss/val_total_loss‚10.419010042664329 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.042845312505960464 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.05000000074505806 particle/accuracy_2_of_5‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.14236323535442352 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.699999988079071 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5370702147483826 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12534311413764954 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 $ _timestamp‚1.784504160124204e+09 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5339846611022949 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6892868876457214 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.748446524143219 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.19432727992534637 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8441795110702515 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5955602526664734 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,jet/accuracy_4_of_4‚0.014682539738714695 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8230767846107483 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.699999988079071 6validation_average_jet_accuracy‚0.3452565670013428 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7346153259277344 jet/accuracy_3_of_5‚1 +jet/accuracy_3_of_3‚0.06594487279653549 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.995820164680481 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3611110746860504 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7477471232414246 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 (particle/accuracy‚0.8467183709144592 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15959374606609344 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1395649015903473 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14688479900360107 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.037718288600444794 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4749999940395355 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8198106288909912 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 jet/accuracy_2_of_5‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37393122911453247 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3636363744735718 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *jet/accuracy_2_of_4‚0.7936181426048279 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9980935454368591 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03679138422012329 *jet/accuracy_1_of_3‚0.9137717485427856 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.625365138053894 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7596637606620789 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.37288087606430054 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4628288745880127 . Purity/2FRt*SRqqt1FBt/FRt_purity‚ 0.4921875 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9969596862792969 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9925780296325684 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.49018582701683044 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872Æjêž,cÈ‚(:Ø ê¾õÒþö¸À$ disk.nvme1n1.out‚3447.4443359375 disk.md0.in‚0$ memory_percent‚98.81641864776611 cpu‚26.570925385553764 network.sent‚ 7.372644e+06 disk.dm-1.in‚0% disk.nvme0n1.out‚14045.5322265625! disk.nvme1c1n1.in‚ 10.48828125) proc.memory.percent‚17.51270294189453 proc.cpu.threads‚17) disk./.usagePercent‚75.80833515812647% disk./.usageGB‚1163.8474655151367 disk.nvme1n1.in‚ 10.48828125 disk.dm-0.in‚ 101.51953125 disk.dm-1.out‚1757.943359375 disk.md0.out‚ 68.17578125 disk.dm-0.out‚ 15589.59375 disk.nvme0c0n1.in‚91.03125' disk.nvme0c0n1.out‚14045.5322265625 disk.nvme0n1.in‚91.03125& disk.nvme1c1n1.out‚3447.4443359375( proc.memory.availableMB‚ 193.91796875 proc.memory.rssMB‚ 2869.28125 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚679418¹—¤~›É‚(: ê¾õÒ̱ŽÁ gpu.0.gpu‚71.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚198.391' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.13033333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏÒG2Ê- _wandbruntime‚2331  _runtime‚2331ˆöKÉËà ' lr-AdamW/pg2‚0.00010795869184682215 trainer/global_step‚5649 % _timestamp‚1.7845041772501547e+09 _runtime‚2331.313971633 _step‚455 ' lr-AdamW/pg1‚0.00010795869184682215—ΕýÎÌÈ _step‚455 ' lr-AdamW/pg1‚0.00010795869184682215 ' lr-AdamW/pg2‚0.00010795869184682215 trainer/global_step‚5649 % _timestamp‚1.7845041772501547e+09 _runtime‚2331.313971633Ç4~]2Í- _wandbruntime‚2331  _runtime‚2331c?Úª¿ÎÒO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 trainer/global_step‚5649 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3978267774976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43029915 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57300915 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24899247 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24899244 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚221916306027520 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚208172136657920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚215115872640176 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚221916200966144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39407782 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚217938021213184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43029913 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14508783 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3965762075374 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100340973 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allv„f%ocated‚ 6618611712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4094161093632 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43029915 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57311058 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3965848223848 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3978179752960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚219081720864024 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43029913 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚217938038252544 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚217938021213184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57311058 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 _runtime‚2331.314034013 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚215115855600816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚221916200966144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚217938038252544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14508538 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚221916306027520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚2122662977515529k…Ý'Ï×O D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 100340973 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4094161093632 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43029915 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57311058 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3965848223848 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3978179752960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚219081720864024 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43029913 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚217938038252544 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚217938021213184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57311058 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 _runtime‚2331.314034013 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚215115855600816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚221916200966144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚217938038252544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14508538 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚221916306027520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚212266297751552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 trainer/global_step‚5649 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3978267774976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43029915 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57300915 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚24899247 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24899244 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚221916306027520 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚208172136657920 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚215115872640176 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚221916200966144 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39407782 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚217938021213184 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43029913 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14508783 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3965762075374 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928ÈË;»2Ð- _wandbruntime‚2331  _runtime‚2331¥"žâ&ÑÜM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 100358206 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4094856300032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24903537 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100348047 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚5649 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚217975387580416 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57310813 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43037236 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24903534 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14510908 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3966524175404 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3978857862656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚221954245443072 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57310813 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199018496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100358206 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43037236 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚208207831815168 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚219119165695394 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845041775161421e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43037234 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚212302688115200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚221954350511616 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39414442 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚212302489096704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚219119268883340 _step‚457 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39414442 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚212302489096704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚219119268883340 _step‚457 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24903537 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 100348047 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚5649 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚217975387580416 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57310813 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43037236 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24903534 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14510908 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 \JDeviceStatsMonitoŒ"@r.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3966524175404 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3978857862656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚221954245443072 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57310813 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199018496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 100358206 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43037236 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚208207831815168 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚219119165695394 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845041775161421e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43037234 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚212302688115200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712É¿Lñ2Ó- _wandbruntime‚2331  _runtime‚2331!Ž uÔ /loss/FRt1/detection_loss‚0.6270905670356124 /loss/FBt2/detection_loss‚0.5823622829962858 _step‚458 2loss/SRqqt2/assignment_loss‚0.6352659781892359 1loss/SRqqt2/detection_loss‚0.6874768408018458 epoch‚7 0loss/FRt1/assignment_loss‚2.5226627293453423 0loss/FRt2/assignment_loss‚1.0807700424682436 /loss/FRt2/detection_loss‚1.4560486478841035 /loss/FBt1/detection_loss‚0.9150418113942058 trainer/global_step‚5649 % _timestamp‚1.7845041775171452e+09 _runtime‚2331.579973047 1loss/FBt2/assignment_loss‚0.05113830897771088 1loss/SRqqt1/detection_loss‚1.1229310439258908 &loss/total_loss‚10.363883646109592 1loss/SRqqt1/assignment_loss‚0.665165421625623 1loss/FBt1/assignment_loss‚0.01792997146549169©†œj Õ† epoch‚7 0loss/FRt1/assignment_loss‚2.5226627293453423 0loss/FRt2/assignment_loss‚1.0807700424682436 /loss/FRt2/detection_loss‚1.4560486478841035 /loss/FBt1/detection_loss‚0.9150418113942058 trainer/global_step‚5649 % _timestamp‚1.7845041775171452e+09 _runtime‚2331.579973047 1loss/FBt2/assignment_loss‚0.05113830897771088 1loss/SRqqt1/detection_loss‚1.1229310439258908 &loss/total_loss‚10.363883646109592 1loss/SRqqt1/assignment_loss‚0.665165421625623 1loss/FBt1/assignment_loss‚0.01792997146549169 /loss/FRt1/detection_loss‚0.6270905670356124 /loss/FBt2/detection_loss‚0.5823622829962858 _step‚458 2loss/SRqqt2/assignment_loss‚0.6352659781892359 1loss/SRqqt2/detection_loss‚0.6874768408018458ÊöbgÖ‚(:Ü ù¾õÒÔ±ö¿$ disk./.usageGB‚1163.878734588623 disk.dm-1.in‚0( disk.nvme0c0n1.out‚14068.19775390625 disk.nvme0n1.in‚91.03125 disk.nvme1n1.in‚ 10.48828125 disk.dm-0.in‚ 101.51953125& disk.nvme0n1.out‚14068.19775390625 disk.md0.in‚0$ memory_percent‚98.81677627563477 cpu‚26.566662550627488 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚91.03125& disk.nvme1c1n1.out‚3513.1142578125$ disk.nvme1n1.out‚3513.1142578125& proc.memory.availableMB‚ 193.859375# disk.dm-0.out‚15612.08447265625! disk.dm-1.out‚1823.1728515625! disk.nvme1c1n1.in‚ 10.48828125 disk.md0.out‚ 68.453125 proc.memory.rssMB‚ 2869.28125) proc.memory.percent‚17.51270294189453 proc.cpu.threads‚17 network.sent‚ 7.443559e+06 network.recv‚683674) disk./.usagePercent‚75.81037189960091ä=!œ×‚(:‘ ù¾õÒ±ãÀ gpu.0.gpu‚55.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚175.075' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚58.358333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Eÿm¨2Ø- _wandbruntime‚2350  _runtime‚2350G»=zÉÙà ' lr-AdamW/pg1‚0.00010787337780803394 ' lr-AdamW/pg2‚0.00010787337780803394 trainer/global_step‚5699 % _timestamp‚1.7845041968453202e+09 _runtime‚2350.909566638 _step‚4595¸ÇBÎÚÈ ' lr-AdamW/pg1‚0.00010787337780803394 ' lr-AdamW/pg2‚0.00010787337780803394 trainer/global_step‚5699 % _timestamp‚1.7845041968453202e+09 _runtime‚2350.909566638 _step‚459Ë1ˆôâ2Û- _wandbruntime‚2350  _runtime‚2350Gºw?Ø'ÜÒO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4012117494784 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57807810 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14632200 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚223818457069568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25113747 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚219806339574784 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57797667 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚214085841271808 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57797667 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚216959476028176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4129145777152 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚223818562130944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4012205516800 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚220959153727432 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3999677699256 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57807810 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚216959458988816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43395963 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14632441 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚214086040297472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5699 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43395963 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4012205516800 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199025664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4128971655168 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚209956894520320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43395965 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚219806356614144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚223818457069568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚223818562130944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4012117494784 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39746188 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39745944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 _runtime‚2350.909629428 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199025664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4128971655168 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚209956894520320 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43395965 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚219806356614144 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚223818457069568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚223818562130944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4012117494784 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39746188 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39745944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 _runtime‚2350.909629428 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57807810 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14632200 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚223818457069568 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25113747 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚219806339574784 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57797667 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚214085841271808 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚57797667 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚216959476028176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4129145777152 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚223818562130944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4012205516800 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚220959153727432 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3999677699256 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57807810 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚216959458988816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43395963 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14632441 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚214086040297472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚5699 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43395963 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4012205516800 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚214122440446464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚209992589677568 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57807565 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43403284 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101221008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4000267502226 % _timestamp‚1.7845041971111982e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚219843722981376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4129850768896 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚219843722981376 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101221008 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101210849 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43403284 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚216996348095936 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚216996331056576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚223856606615040 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39752676 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25118034 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚219843705942016 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4012883633664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4012883633664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4012795604480 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚223856501546496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101210849 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 trainer/global_step‚5699 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4012795604480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43403286 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚214122228845056 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57807565 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14634915 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚209992564773888 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14634642 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚220996701746748 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0Cwdùç&àáM P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚57807565 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14634915 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚276 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚209992564773888 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14634642 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚220996701746748 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚223856606615040 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43403286 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 _runtime‚2351.174476008 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚214122440446464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚209992589677568 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚57807565 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43403284 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101221008 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4000267502226 % _timestamp‚1.7845041971111982e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚219843722981376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4129850768896 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚219843722981376 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101221008 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101210849 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43403284 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚216996348095936 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚216996331056576 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚223856606615040 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚273 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39752676 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25118034 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚219843705942016 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4012883633664 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4012883633664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4012795604480 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚223856501546496 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101210849 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 trainer/global_step‚5699 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4012795604480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43403286 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚214122228845056Í^Ôæ2á- _wandbruntime‚2351  _runtime‚2351YGþâþ epoch‚7 /loss/FRt1/assignment_loss‚2.656349447395267 2loss/SRqqt1/assignment_loss‚0.6987658678273136 /loss/FRt2/detection_loss‚1.5900925978101972 &loss/total_loss‚11.421921159863645 1loss/FBt2/assignment_loss‚0.07976189664146822 0loss/SRqqt1/detection_loss‚1.036024169123673 .loss/FBt1/detection_loss‚1.378596394930197 trainer/global_step‚5699 % _timestamp‚1.7845041971122146e+09 0loss/FBt1/assignment_loss‚0.1086573563926426 1loss/SRqqt2/detection_loss‚0.7323166447026006 /loss/FBt2/detection_loss‚0.6481108837936707 _runtime‚2351.175035169 _step‚462 0loss/FRt2/assignment_loss‚1.1870285062654782 2loss/SRqqt2/assignment_loss‚0.7651048378763259 /loss/FRt1/detection_loss‚0.5411125571048113zâ É ム0loss/FRt2/assignment_loss‚1.1870285062654782 2loss/SRqqt2/assignment_loss‚0.7651048378763259 /loss/FRt1/detection_loss‚0.5411125571048113 epoch‚7 /loss/FRt1/assignment_loss‚2.656349447395267 2loss/SRqqt1/assignment_loss‚0.6987658678273136 /loss/FRt2/detection_loss‚1.5900925978101972 &loss/total_loss‚11.421921159863645 1loss/FBt2/assignment_loss‚0.07976189664146822 0loss/SRqqt1/detection_loss‚1.036024169123673 .loss/FBt1/detection_loss‚1.378596394930197 trainer/global_step‚5699 % _timestamp‚1.7845041971122146e+09 0loss/FBt1/assignment_loss‚0.1086573563926426 1loss/SRqqt2/detection_loss‚0.7323166447026006 /loss/FBt2/detection_loss‚0.6481108837936707 _runtime‚2351.175035169 _step‚462·ÿÐhä‚(:Ý ˆ¿õÒ‚Í¿ network.sent‚ 7.491859e+06 disk.rbd0.out‚0( disk.nvme0c0n1.out‚14083.34130859375& disk.nvme0n1.out‚14083.34130859375 disk.md0.in‚0$ memory_percent‚98.81696701049805& proc.memory.availableMB‚ 193.828125 proc.memory.rssMB‚ 2869.28125% disk./.usageGB‚1163.8789138793945 disk.dm-0.in‚ 101.51953125# disk.dm-0.out‚15626.69482421875 disk.dm-1.in‚0 cpu‚26.562343581799393 network.recv‚687979 disk.md0.out‚ 68.984375! disk.dm-1.out‚1823.2783203125) proc.memory.percent‚17.51270294189453 proc.cpu.threads‚17) disk./.usagePercent‚75.81038357787946 disk.rbd0.in‚0 disk.nvme0c0n1.in‚91.03125 disk.nvme0n1.in‚91.03125! disk.nvme1c1n1.in‚ 10.48828125& disk.nvme1c1n1.out‚3513.8310546875 disk.nvme1n1.in‚ 10.48828125$ disk.nvme1n1.out‚3513.8310546875d‡•›å‚(: ˆ¿õÒή¸À gpu.0.gpu‚53.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚198.561' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.18700000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^Y0Ñhæ‚(:Ý —¿õÒÊÈî¿ network.recv‚692078& proc.memory.availableMB‚ 193.828125 cpu‚26.558484741134414 proc.cpu.threads‚17 network.sent‚ 7.555221e+06% disk./.usageGB‚1164.2001914978027 disk.dm-0.in‚ 101.51953125 disk.nvme0n1.in‚91.03125! disk.nvme1c1n1.in‚ 10.48828125$ disk.nvme1n1.out‚3597.0517578125) disk./.usagePercent‚75.83131030762435 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚91.03125( disk.nvme0c0n1.out‚14179.20166015625& disk.nvme1c1n1.out‚3597.0517578125 disk.md0.in‚0 proc.memory.rssMB‚ 2869.28125# disk.dm-0.out‚15804.37255859375 disk.dm-1.out‚ 1823.94140625 disk.rbd0.in‚0& disk.nvme0n1.out‚14179.20166015625 disk.nvme1n1.in‚ 10.48828125 disk.md0.out‚ 69.33203125$ memory_percent‚98.81696701049805) proc.memory.percent‚17.51270294189453W*æQ›ç‚(: —¿õÒ°ê©À gpu.0.gpu‚57.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚202.776' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.59200000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|þ. 2è- _wandbruntime‚2370  _runtime‚2370ˆ>Ééà ' lr-AdamW/pg1‚0.00010778723392902229 ' lr-AdamW/pg2‚0.00010778723392902229 trainer/global_step‚5749 % _timestamp‚1.7845042164287965e+09 _runtime‚2370.492575277 _step‚463¥âúñÎêÈ ' lr-AdamW/pg2‚0.00010778723392902229 trainer/global_step‚5749 % _timestamp‚1.7845042164287965e+09 _runtime‚2370.492575277 _step‚463 ' lr-AdamW/pg1‚0.00010778723392902229ω·F2ë- _wandbruntime‚2370  _runtime‚2370ã»tCØ'ìÒO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43762015 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4033421026190 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚225720713172992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218803079416176 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40083984 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43762013 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 trainer/global_step‚5749 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211741627479040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4033507174664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43762015 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚222836586590840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚225720818234368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4046055236608 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4046055236608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚222836483403006 % _timestamp‚1.7845042164294736e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4046143258624 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚225720713172992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218803079416176 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40083984 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚43762013 G>DeviceStatsMonitor.on_train_batch_start–÷—üÇ/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 trainer/global_step‚5749 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211741627479040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4033507174664 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43762015 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚222836586590840 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚225720818234368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4046055236608 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4046055236608 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚222836483403006 % _timestamp‚1.7845042164294736e+09 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4046143258624 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374Д™2î- _wandbruntime‚2370  _runtime‚2370 ®â&ïÜM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚215942168707584 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4164628178432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43769334 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 _step‚465 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102083810 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4164821167616 % _timestamp‚1.7845042166950471e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚225758862718464 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58314474 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚222874031422210 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43769336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14758242 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚225758757649920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5749 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25332537 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚218839951483936 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102083810 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58304317 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4046821375488 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102073651 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4034096977634 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚218839934444576 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚221712041342976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43769336 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40090516 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _runtime‚2370.758321071 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102073651 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚211777347539968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211777322636288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚222874134610156 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4046821375488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464—¶E|ç&ðáM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚263 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25332534 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5749 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25332537 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚218839951483936 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102083810 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58304317 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4046821375488 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 102073651 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4034096977634 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚218839934444576 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚221712041342976 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43769336 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40090516 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _runtime‚2370.758321071 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 102073651 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚211777347539968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211777322636288 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚222874134610156 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4046821375488 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚215942168707584 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4164628178432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43769334 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 _step‚465 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102083810 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4164821167616 % _timestamp‚1.7845042166950471e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚225758862718464 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58314474 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚222874031422210 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43769336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14758242 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚225758757649920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40423160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚224714019454248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102919234 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4080081000448 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚220646665764816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44128063 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚227623074337792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227622969276416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4080081000448 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40422879 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚213526410245120 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚227623074337792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58791171 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25542747 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚223542976297984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚224713916266414 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚223542976297984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58791171 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14880413 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚217725258565632 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25542744 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58801314 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44128065 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4067250501598£05µÜ'ûÖO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4080081000448 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40422879 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚213526410245120 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚227623074337792 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚278 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58791171 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25542747 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚223542976297984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚224713916266414 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚223542976297984 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58791171 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14880413 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚217725258565632 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25542744 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚58801314 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44128065 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4067250501598 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚227622969276416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14880135 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215802880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 $ _timestamp‚1.784504235740355e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚40423160 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚224714019454248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 102919234 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4080081000448 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚220646665764816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44128063 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚227623074337792 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚281 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227622969276416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712Ô°µ.‰2ü- _wandbruntime‚2390  _runtime‚2390åqüá&ýÛM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44135384 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚217761653634560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102946612 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚220683554871936 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4068012601628 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44135384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚227661118821888 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 $ _timestamp‚1.784504236008024e+09 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58811226 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 _step‚469 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58801069 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4080671088128 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40429708 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25547034 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚224751567473564 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58811226 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚223580359704576 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4080759117312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14882674 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4199573135872 _runtime‚2390.071298289 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚227661118821888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚227661013753344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40429975 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚217761863138816 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚223580342665216 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚223580359704576 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4080759117312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25547037 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14882938 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚224751464285618 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4067926453042 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102946612 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚223580342665216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4080671088128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44135386 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104Ì—ÔÎþàM E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44135384 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚227661118821888 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 $ _timestamp‚1.784504236008024e+09 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58811226 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 _step‚469 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58801069 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.cur¸·¦rent‚10157 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4080671088128 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40429708 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25547034 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚224751567473564 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58811226 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚223580359704576 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4080759117312 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14882674 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4199573135872 _runtime‚2390.071298289 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚227661118821888 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚227661013753344 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40429975 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚217761863138816 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚223580342665216 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚223580359704576 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4080759117312 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25547037 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14882938 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚224751464285618 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4067926453042 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 102946612 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚223580342665216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4080671088128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚44135386 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44135384 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚217761653634560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 102946612 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚220683554871936 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4068012601628 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256ÕÄ·Ã2ÿ- _wandbruntime‚2390  _runtime‚2390jIñ>€ÿ 1loss/FBt1/assignment_loss‚0.21673299964934037 -loss/FRt2/detection_loss‚1.49938654925066 0loss/FBt2/detection_loss‚0.46215701022332434 %loss/total_loss‚10.18275796882957 _runtime‚ 2390.07189864 _step‚470 0loss/FRt2/assignment_loss‚1.0798712097868628 1loss/FBt2/assignment_loss‚0.03824857310407624 /loss/FBt1/detection_loss‚0.6922586948975272 trainer/global_step‚5799 0loss/FRt1/assignment_loss‚2.5411135878354782 /loss/FRt1/detection_loss‚0.5058748915082163 1loss/SRqqt1/detection_loss‚1.0476102828614267 % _timestamp‚1.7845042360090425e+09 2loss/SRqqt1/assignment_loss‚0.7763674107290427 2loss/SRqqt2/assignment_loss‚0.6598215126774056 1loss/SRqqt2/detection_loss‚0.6633152463062092 epoch‚7š „ 0loss/FRt1/assignment_loss‚2.5411135878354782 /loss/FRt1/detection_loss‚0.5058748915082163 1loss/SRqqt1/detection_loss‚1.0476102828614267 % _timestamp‚1.7845042360090425e+09 2loss/SRqqt1/assignment_loss‚0.7763674107290427 2loss/SRqqt2/assignment_loss‚0.6598215126774056 1loss/SRqqt2/detection_loss‚0.6633152463062092 epoch‚7 1loss/FBt1/assignment_loss‚0.21673299964934037 -loss/FRt2/detection_loss‚1.49938654925066 0loss/FBt2/detection_loss‚0.46215701022332434 %loss/total_loss‚10.18275796882957 _runtime‚ 2390.07189864 _step‚470 0loss/FRt2/assignment_loss‚1.0798712097868628 1loss/FBt2/assignment_loss‚0.03824857310407624 /loss/FBt1/detection_loss‚0.6922586948975272 trainer/global_step‚5799Öôaºi‚‚(:Þ µ¿õÒ¥ÇÀ proc.cpu.threads‚17 disk.nvme0c0n1.in‚91.03125 disk.nvme0n1.in‚91.03125 proc.memory.rssMB‚ 2869.28125 cpu‚26.550606722072967 network.sent‚ 7.633192e+06 network.recv‚698716 disk.md0.out‚ 70.35546875 disk.rbd0.out‚0% disk.nvme0n1.out‚14320.2763671875! disk.nvme1c1n1.in‚ 10.48828125 disk.nvme1n1.in‚ 10.48828125$ disk.nvme1n1.out‚3603.6708984375) disk./.usagePercent‚75.82994345208563 disk.dm-0.in‚ 101.51953125 disk.dm-1.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚14320.2763671875$ memory_percent‚98.81761074066162( proc.memory.availableMB‚ 193.72265625) proc.memory.percent‚17.51270294189453% disk./.usageGB‚1164.1792068481445 disk.md0.in‚0" disk.dm-0.out‚15949.5224609375! disk.dm-1.out‚1824.2919921875& disk.nvme1c1n1.out‚3603.6708984375Mw·½œƒ‚(:‘ µ¿õÒœ®µÀ gpu.0.gpu‚88.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚122.53' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚40.843333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸Ú™2„- _wandbruntime‚2409  _runtime‚2409õy2È…Â trainer/global_step‚5849 $ _timestamp‚1.784504255335618e+09 _runtime‚2409.398756493 _step‚471 ' lr-AdamW/pg1‚0.00010761246218267444 ' lr-AdamW/pg2‚0.00010761246218267444—ï4òꠂ ' lr-AdamW/pg1‚0.00010761246218267444 ' lr-AdamW/pg2‚0.00010761246218267444 trainer/global_step‚5849 $ _timestamp‚1.784504255335618e+09 _runtime‚2409.398756493 _step‚471×{ÏCÓ2‡- _wandbruntime‚2409  _runtime‚2409¨Y­××'ˆÑO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44494115 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4114018742272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚219545175327744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59287923 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚219544963719168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4233820515328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44494113 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚225411294659584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚215311168107520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚215311143203840 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚226591452317656 _step‚472 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚229525225379840 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15003875 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚229525330441216 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59287923 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚225411311698944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717É›Ü'‰ÖO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚225411294659584 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚215311168107520 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚215311143203840 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚226591452317656 _step‚472 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚229525225379840 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15003875 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚229525330441216 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59287923 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚225411311698944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44494115 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4114018742272 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚219545175327744 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚59287923 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚219544963719168 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4233820515328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44494113 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4114608829952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚226629000336972 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚225448661026816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44501434 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4101755928450 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44501436 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4234703845888 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚225448678066176 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚2409.664294798 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚229563374925312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103799255 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59297821 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.largÿjìˆe_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59297821 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15006024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚473 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4114696859136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40767558 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚215346838361088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚229563374925312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4114608829952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4114696859136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25761537 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 $ _timestamp‚1.784504255601006e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103809414 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚226628897149026 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103799255 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚225448678066176 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103809414 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚219581567110656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25761534 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚5849 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚219581365995008 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4101842077036 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44501434 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚2409.664294798 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚229563374925312 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 103799255 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59297821 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59297821 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15006024 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚473 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4114696859136 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40767558 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚215346838361088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚229563374925312 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4114608829952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4114696859136 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25761537 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 $ _timestamp‚1.784504255601006e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103809414 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚226628897149026 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103799255 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚225448678066176 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103809414 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚219581567110656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25761534 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚5849 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚219581365995008 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4101842077036 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44501434 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4114608829952 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚226629000336972 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚225448661026816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44501434 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4101755928450 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44501436 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4234703845888Ùúe·2- _wandbruntime‚2409  _runtime‚2409–뎂 trainer/global_step‚5849 _step‚474 1loss/FBt1/assignment_loss‚0.10944561779930202 % _timestamp‚1.7845042556020133e+09 _runtime‚2409.665163359 /loss/FBt2/detection_loss‚0.6147474444045101 %loss/total_loss‚10.31647606857035 0loss/FRt1/assignment_loss‚2.4430828919314376 2loss/SRqqt2/assignment_loss‚0.6929553890516374 3loss/FBt2/assignment_loss‚0.0026751262539638616 1loss/SRqqt1/detection_loss‚0.9575685816263074 /loss/FRt1/detection_loss‚0.5311150514943151 /loss/FRt2/detection_loss‚1.4472340351623711 epoch‚7 0loss/FRt2/assignment_loss‚1.0898881069852109 2loss/SRqqt1/assignment_loss‚0.8220101122288108 1loss/SRqqt2/detection_loss‚0.6562371426474675 .loss/FBt1/detection_loss‚0.949516568985016Ü”° ‡ 1loss/FBt1/assignment_loss‚0.10944561779930202 % _timestamp‚1.7845042556020133e+09 _runtime‚2409.665163359 /loss/FBt2/detection_loss‚0.6147474444045101 %loss/total_loss‚10.31647606857035 0loss/FRt1/assignment_loss‚2.4430828919314376 2loss/SRqqt2/assignment_loss‚0.6929553890516374 3loss/FBt2/assignment_loss‚0.0026751262539638616 1loss/SRqqt1/detection_loss‚0.9575685816263074 /loss/FRt1/detection_loss‚0.5311150514943151 /loss/FRt2/detection_loss‚1.4472340351623711 epoch‚7 0loss/FRt2/assignment_loss‚1.0898881069852109 2loss/SRqqt1/assignment_loss‚0.8220101122288108 1loss/SRqqt2/detection_loss‚0.6562371426474675 .loss/FBt1/detection_loss‚0.949516568985016 trainer/global_step‚5849 _step‚474ÚȶñR]‚(:Ò Ä¿õÒ¼ú¿ network.sent‚ 7.695329e+06! disk.dm-1.out‚1824.4169921875 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme1c1n1.in‚ 10.48828125 disk.nvme1n1.in‚ 10.48828125 disk.md0.in‚0) proc.memory.percent‚17.51270294189453) disk./.usagePercent‚75.82996084526644% disk./.usageGB‚1164.1794738769531 disk.dm-1.in‚0 disk.nvme0n1.in‚91.03125# disk.nvme0n1.out‚14331.75390625$ disk.nvme1c1n1.out‚ 3604.69140625" disk.nvme1n1.out‚ 3604.69140625 disk.dm-0.in‚ 101.51953125 disk.nvme0c0n1.in‚91.03125" disk.dm-0.out‚15960.6748046875$ memory_percent‚98.81782531738281$ proc.memory.availableMB‚193.6875 proc.cpu.threads‚17 network.recv‚703003% disk.nvme0c0n1.out‚14331.75390625 disk.md0.out‚ 70.93359375 proc.memory.rssMB‚ 2869.28125 cpu‚26.546695377283886}j›‘‚(: Ä¿õÒº¤§À gpu.0.gpu‚50.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚200.836' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.94533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µâÅP2’- _wandbruntime‚2428  _runtime‚2428ĤÉ“Ã ' lr-AdamW/pg1‚0.00010752383711466251 ' lr-AdamW/pg2‚0.00010752383711466251 trainer/global_step‚5899 % _timestamp‚1.7845042749049332e+09 _runtime‚2428.969589032 _step‚475FGþlÎ”È ' lr-AdamW/pg1‚0.00010752383711466251 ' lr-AdamW/pg2‚0.00010752383711466251 trainer/global_step‚5899 % _timestamp‚1.7845042749049332e+09 _runtime‚2428.969589032 _step‚475ÛN#¯J2•- _wandbruntime‚2428  _runtime‚2428ÛŽ/wØ'–ÒO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚231427586544640 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4269018626560 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44860165 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚228468781993230 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59784675 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4147868462080 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4268838213120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44860165 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41098801 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚221364944596480 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚231427481483264 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4147956484096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚224333889580176 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104654983 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚5899 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104644838 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚217095901066240 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59794818 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59794818 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25971744 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚231427481483264 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚476 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4147868462080 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44860163 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44860163 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚227279613021184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚231427586544640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4134995600888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚227279630060544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚228468885181064ûLz«. —×O =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚224333872540816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _runtime‚2428.969646092 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚476 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4147868462080 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44860163 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44860163 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚227279613021184 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚231427586544640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚Ó¬rŽ¯12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4134995600888 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚227279630060544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚228468885181064 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚231427586544640 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4269018626560 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44860165 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚228468781993230 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59784675 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4147868462080 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4268838213120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44860165 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41098801 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚221364944596480 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚231427481483264 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4147956484096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚224333889580176 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 104654983 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚5899 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104644838 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚217095901066240 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59794818 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚59794818 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚25971744 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚231427481483264 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4148546571776 trainer/global_step‚5899 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104662057 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7845042751719873e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59794573 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41105217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚227316996427776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚231465631028736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚224370744608576 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4148634600960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41105467 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚221401336639488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚228506330012434 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚231465631028736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 196921344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104672216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚227316979388416 _step‚477 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚228506433200380 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4135585403858 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _runtime‚2429.235269258À2®|ç&šáM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4148634600960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41105467 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚221401336639488 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚228506330012434 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚231465631028736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 196921344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104672216 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚227316979388416 _step‚477 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚228506433200380 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4135585403858 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _runtime‚2429.235269258 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44867486 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44867484 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25976037 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚231465525960192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚224370761647936 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4148546571776 trainer/global_step‚5899 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104662057 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7845042751719873e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚59794573 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41105217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚227316996427776 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚231465631028736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚224370744608576 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482Ýh«l2›- _wandbruntime‚2429  _runtime‚2429öƒÞ“œ 2loss/FBt2/assignment_loss‚0.003643186435596135 /loss/FRt1/detection_loss‚0.5630137815570486 1loss/SRqqt1/detection_loss‚1.2008168042088898 .loss/FBt1/detection_loss‚1.359739967717936 _step‚478 1loss/SRqqt2/assignment_loss‚0.611240680586742 /loss/FRt2/detection_loss‚1.4583091267288244 /loss/FBt2/detection_loss‚0.5852891373047426 trainer/global_step‚5899 _runtime‚ 2429.23582584 &loss/total_loss‚10.694258549898816 0loss/FRt1/assignment_loss‚2.4803300416597756 2loss/SRqqt1/assignment_loss‚0.7317452072880111 1loss/SRqqt2/detection_loss‚0.5997095402994854 2loss/FBt1/assignment_loss‚0.003029692482114528 epoch‚7 % _timestamp‚1.7845042751729949e+09 0loss/FRt2/assignment_loss‚1.0973913836296492Qjú› † 2loss/FBt1/assignment_loss‚0.003029692482114528 epoch‚7 % _timestamp‚1.7845042751729949e+09 0loss/FRt2/assignment_loss‚1.0973913836296492 2loss/FBt2/assignment_loss‚0.003643186435596135 /loss/FRt1/detection_loss‚0.5630137815570486 1loss/SRqqt1/detection_loss‚1.2008168042088898 .loss/FBt1/detection_loss‚1.359739967717936 _step‚478 1loss/SRqqt2/assignment_loss‚0.611240680586742 /loss/FRt2/detection_loss‚1.4583091267288244 /loss/FBt2/detection_loss‚0.5852891373047426 trainer/global_step‚5899 _runtime‚ 2429.23582584 &loss/total_loss‚10.694258549898816 0loss/FRt1/assignment_loss‚2.4803300416597756 2loss/SRqqt1/assignment_loss‚0.7317452072880111 1loss/SRqqt2/detection_loss‚0.5997095402994854Þ ÑÎ*hž‚(:Ý Ó¿õÒêÆù¿ disk.nvme0c0n1.in‚91.03125' disk.nvme0c0n1.out‚14363.2080078125% disk.nvme0n1.out‚14363.2080078125 disk.nvme1n1.in‚ 10.48828125 disk.md0.in‚0" disk.dm-1.out‚1824.62646484375 disk.rbd0.out‚0) proc.memory.percent‚17.51270294189453( disk./.usagePercent‚75.8381097989563 disk.md0.out‚ 71.34765625 disk.dm-0.in‚ 101.51953125' proc.memory.availableMB‚ 193.4296875 proc.memory.rssMB‚ 2869.28125 cpu‚26.54282884576889 disk.nvme0n1.in‚91.03125$ disk.nvme1n1.out‚3605.9091796875$ memory_percent‚98.81939888000488 proc.cpu.threads‚17 network.sent‚ 7.757502e+06 network.recv‚706892% disk./.usageGB‚1164.3045806884766! disk.nvme1c1n1.in‚ 10.48828125& disk.nvme1c1n1.out‚3605.9091796875# disk.dm-0.out‚15992.24560546875 disk.dm-1.in‚0 disk.rbd0.in‚0É?VŠ›Ÿ‚(: Ó¿õÒÈÿ¡À gpu.0.gpu‚74.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.904' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.30133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›Kdl ‚(:á â¿õÒåæ׿& disk.nvme0n1.out‚14372.23681640625' disk.nvme1c1n1.out‚3606.69189453125" disk.dm-1.out‚1824.70458984375 disk.rbd0.in‚0 disk.rbd0.out‚0) proc.memory.percent‚17.51270294189453 cpu‚26.539015293420107 proc.cpu.threads‚17 network.sent‚ 7.807155e+06 disk.nvme0c0n1.in‚91.03125 disk.nvme1n1.in‚ 10.48828125' proc.memory.availableMB‚ 193.6484375) disk./.usagePercent‚75.83811327759247 disk.nvme0n1.in‚91.03125 disk.dm-0.in‚ 101.51953125# disk.dm-0.out‚16000.68701171875$ memory_percent‚98.81806373596191 proc.memory.rssMB‚ 2869.28125 disk.md0.in‚0 disk.md0.out‚ 71.953125 network.recv‚711415% disk./.usageGB‚1164.3046340942383( disk.nvme0c0n1.out‚14372.23681640625! disk.nvme1c1n1.in‚ 10.48828125% disk.nvme1n1.out‚3606.69189453125 disk.dm-1.in‚0f^cΡ‚(:… â¿õÒŠü”À gpu.0.gpu‚11.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.528' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.176 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03"‚2¢- _wandbruntime‚2448  _runtime‚2448B~D¬É£Ã % _timestamp‚1.7845042944681792e+09 _runtime‚2448.531334594 _step‚479 ' lr-AdamW/pg1‚0.00010743438780503083 ' lr-AdamW/pg2‚0.00010743438780503083 trainer/global_step‚5949~t Î¤È _step‚479 ' lr-AdamW/pg1‚0.00010743438780503083 ' lr-AdamW/pg2‚0.00010743438780503083 trainer/global_step‚5949 % _timestamp‚1.7845042944681792e+09 _runtime‚2448.531334594ßÈãy˜2¥- _wandbruntime‚2448  _runtime‚2448+þæͧ¦ÒO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚218880683832320 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚218880658928640 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.ë»qß1&small_pool.freed‚ 4181806203904 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚230346214856638 _step‚480 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚233329737586688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚226177475928816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 % _timestamp‚1.7845042944691622e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26186247 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15250773 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚229147948422144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4181806203904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15250513 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚230346318044472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45226215 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 105507640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60291570 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚230346214856638 _step‚480 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚233329737586688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚226177475928816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 % _timestamp‚1.7845042944691622e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26186247 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15250773 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚229147948422144 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4181806203904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15250513 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚230346318044472 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136àJ5æ2¨- _wandbruntime‚2448  _runtime‚2448‰fâ&©ÜM E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚481 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60291325 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚218916354085888 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60291325 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚233367887132160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚230383762875842 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚229185314789376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4304700672000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 _runtime‚2448.800096912 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4182572342784 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4182484313600 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚233367782063616 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60301482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚223221079661568 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚223220872254464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚218916378989568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26190537 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4304518168576 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚229185297750016 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41443556 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105535018 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4169414879266 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45233536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45233534 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712Î=aq» ªáM SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚481 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60291325 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚218916354085888 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60291325 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚233367887132160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚230383762875842 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚229185314789376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4304700672000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 _runtime‚2448.800096912 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4182572342784 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4182484313600 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚233367782063616 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60301482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚223221079661568 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚223220872254464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚218916378989568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26190537 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4304518168576 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚229185297750016 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41443556 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 105535018 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4169414879266 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45233536 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45233534 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚267á>`¬¬2«- _wandbruntime‚2448  _runtime‚2448hˆìx ¬„ /loss/FBt2/detection_loss‚0.6840030130252436 % _timestamp‚1.7845042947378376e+09 2loss/SRqqt1/assignment_loss‚0.7467542384480943 2loss/SRqqt2/assignment_loss‚0.6951124598248928 3loss/FBt2/assignment_loss‚0.0034548188446655737 1loss/SRqqt1/detection_loss‚1.1203922275338827 /loss/FBt1/detection_loss‚1.7870283983147728 trainer/global_step‚5949 _step‚482 /loss/FRt1/assignment_loss‚2.650242909574017 epoch‚7 _runtime‚2448.800712854 &loss/total_loss‚11.598964639999135 /loss/FRt2/assignment_loss‚1.062354749821602 3loss/FBt1/assignment_loss‚0.0023062043134826934 /loss/FRt1/detection_loss‚0.5249696791857018 /loss/FRt2/detection_loss‚1.6261726348839225 1loss/SRqqt2/detection_loss‚0.6961733062288583ò!@­‰ 2loss/SRqqt2/assignment_loss‚0.6951124598248928 3loss/FBt2/assignment_loss‚0.0034548188446655737 1loss/SRqqt1/detection_loss‚1.1203922275338827 /loss/FBt1/detection_loss‚1.7870283983147728 trainer/global_step‚5949 _step‚482 /loss/FRt1/assignment_loss‚2.650242909574017 epoch‚7 _runtime‚2448.800712854 &loss/total_loss‚11.598964639999135 /loss/FRt2/assignment_loss‚1.062354749821602 3loss/FBt1/assignment_loss‚0.0023062043134826934 /loss/FRt1/detection_loss‚0.5249696791857018 /loss/FRt2/detection_loss‚1.6261726348839225 1loss/SRqqt2/detection_loss‚0.6961733062288583 /loss/FBt2/detection_loss‚0.6840030130252436 % _timestamp‚1.7845042947378376e+09 2loss/SRqqt1/assignment_loss‚0.7467542384480943âeqöm®‚(:â ñ¿õÒäÃÚÁ proc.memory.rssMB‚ 2869.28125 proc.cpu.threads‚17 disk.dm-0.in‚ 101.51953125 disk.nvme0n1.in‚91.03125$ memory_percent‚98.81854057312012) proc.memory.percent‚17.51270294189453 network.recv‚714430) disk./.usagePercent‚75.83814433684394 disk.md0.in‚0 disk.dm-1.in‚0" disk.dm-1.out‚1824.88525390625& disk.nvme0n1.out‚14379.03466796875! disk.nvme1c1n1.in‚ 10.48828125 disk.nvme1n1.in‚ 10.48828125 network.sent‚ 7.813237e+06 disk.md0.out‚ 72.3828125# disk.dm-0.out‚16007.11083984375 disk.nvme0c0n1.in‚91.03125( disk.nvme0c0n1.out‚14379.03466796875% disk.nvme1n1.out‚3607.41845703125' proc.memory.availableMB‚ 193.5703125 cpu‚26.535196618028987% disk./.usageGB‚1164.3051109313965 disk.rbd0.in‚0 disk.rbd0.out‚0' disk.nvme1c1n1.out‚3607.41845703125Úk‡"¯‚(:… ñ¿õÒºëóÀ gpu.0.gpu‚46.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚112.464' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.488 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f¨¿å2°- _wandbruntime‚2467  _runtime‚2467œ@%ȱ _step‚483 ' lr-AdamW/pg1‚0.00010734411568649167 ' lr-AdamW/pg2‚0.00010734411568649167 trainer/global_step‚5999 $ _timestamp‚1.784504313706945e+09 _runtime‚2467.770127734¢^çüÍ²Ç ' lr-AdamW/pg1‚0.00010734411568649167 ' lr-AdamW/pg2‚0.00010734411568649167 trainer/global_step‚5999 $ _timestamp‚1.784504313706945e+09 _runtime‚2467.770127734 _step‚483ãß&¯2³- _wandbruntime‚2467  _runtime‚2467oAXçØ'´ÒO % _timestamp‚1.7845043137078993e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60778179 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45592263 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60778179 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚231016266783744 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚232223647720046 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚228021096356176 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41775138 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106370442 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15374394 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4202654551704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚235232098751488 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4215743945728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26400747 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚220665441694720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4215743945728 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚231016249744384 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚225004394326528 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60788322 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235231993690112 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45592265 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41775396 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235231993690112 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106380587 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45592263 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚60788322 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚235232098751488 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚232223750907880 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚220665416791040 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚232223750907880 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚220665416791040 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15374394 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4202654551704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚235232098751488 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4215743945728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26400747 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚220665441694720 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4215743945728 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚231016249744384 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚225004394326528 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60788322 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235231993690112 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45592265 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚41775396 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235231993690112 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448ä>+­œ2¶- _wandbruntime‚2468  _runtime‚2468h7šJ·ÜM % _timestamp‚1.7845043142873604e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4216422055424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15376881 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚220701111948288 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚228057951384576 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45599586 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15377132 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.o?a /˜ n_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106397820 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚232261195739250 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚231053633150976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚220701136851968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41782169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106387661 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4216510084608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41781915 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26405034 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚485 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106397820 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4203330503260 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚225040789074944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60798234 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26405037 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4339469719552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45599584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚231053633150976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚235270143235584 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚231053616111616 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚235270143235584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4203244354674 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60788077 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4216510084608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4339652222976 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60798234 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚232261298927196 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106387661 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚225040581667840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0õtS¸ç&¸áM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚225040789074944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60798234 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26405037 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4339469719552 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45599584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚231053633150976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚235270143235584 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚231053616111616 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚235270143235584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4203244354674 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60788077 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4216510084608 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4339652222976 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60798234 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚232261298927196 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 106387661 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚225040581667840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845043142873604e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4216422055424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15376881 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚220701111948288 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚228057951384576 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45599586 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15377132 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106397820 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚232261195739250 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚231053633150976 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚220701136851968 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41782169 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106387661 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4216510084608 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41781915 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26405034 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚485 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106397820 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4203330503260åÛ‡ '2¹- _wandbruntime‚2468  _runtime‚2468DJкþ 2loss/SRqqt1/assignment_loss‚0.7093422848075254 /loss/FRt1/detection_loss‚0.5695113448411823 /loss/FBt2/detection_loss‚0.8408854279304081 epoch‚7 trainer/global_step‚5999 $ _timestamp‚1.784504314288464e+09 /loss/FRt1/assignment_loss‚2.598711707662878 1loss/FBt1/assignment_loss‚0.15894464351547938 2loss/FBt2/assignment_loss‚0.014529041129172407 1loss/SRqqt1/detection_loss‚1.0345188207666642 _runtime‚ 2468.3517748 2loss/SRqqt2/assignment_loss‚0.6304162195005752 .loss/FRt2/detection_loss‚1.502618098421861 /loss/FBt1/detection_loss‚0.7103449090746625 0loss/FRt2/assignment_loss‚1.0609578566247249 1loss/SRqqt2/detection_loss‚0.6812461946823362 &loss/total_loss‚10.512026548957468 _step‚486j¤a¦ »ƒ 2loss/FBt2/assignment_loss‚0.014529041129172407 1loss/SRqqt1/detection_loss‚1.0345188207666642 _runtime‚ 2468.3517748 2loss/SRqqt2/assignment_loss‚0.6304162195005752 .loss/FRt2/detection_loss‚1.502618098421861 /loss/FBt1/detection_loss‚0.7103449090746625 0loss/FRt2/assignment_loss‚1.0609578566247249 1loss/SRqqt2/detection_loss‚0.6812461946823362 &loss/total_loss‚10.512026548957468 _step‚486 2loss/SRqqt1/assignment_loss‚0.7093422848075254 /loss/FRt1/detection_loss‚0.5695113448411823 /loss/FBt2/detection_loss‚0.8408854279304081 epoch‚7 trainer/global_step‚5999 $ _timestamp‚1.784504314288464e+09 /loss/FRt1/assignment_loss‚2.598711707662878 1loss/FBt1/assignment_loss‚0.15894464351547938æ¸z6l¼‚(:á €ÀõÒÄ´†À% disk./.usageGB‚1164.0681686401367' disk.nvme1c1n1.out‚3608.04736328125 disk.nvme1n1.in‚ 10.48828125 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.md0.out‚ 72.68359375 disk.dm-0.in‚ 101.51953125 disk.rbd0.out‚0 disk.nvme0c0n1.in‚91.03125 cpu‚26.531405701483926! disk.nvme1c1n1.in‚ 10.48828125# disk.dm-0.out‚16033.15087890625$ memory_percent‚98.81877899169922 proc.memory.rssMB‚ 2869.28125 proc.cpu.threads‚17) disk./.usagePercent‚75.82271087055116% disk.nvme1n1.out‚3608.04736328125" disk.dm-1.out‚1824.97900390625( disk.nvme0c0n1.out‚14405.18798828125 disk.nvme0n1.in‚91.03125& disk.nvme0n1.out‚14405.18798828125% proc.memory.availableMB‚ 193.53125) proc.memory.percent‚17.51270294189453 network.sent‚ 7.887273e+06 network.recv‚718744%-,!›½‚(: €ÀõҘƮÀ gpu.0.gpu‚3.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.782' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.260666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0® &œ2¾- _wandbruntime‚2487  _runtime‚2487«ÿ`jÅ¿¿ _runtime‚ 2487.21141487 _step‚487 & lr-AdamW/pg1‚0.0001072530222049362 & lr-AdamW/pg2‚0.0001072530222049362 trainer/global_step‚6049 $ _timestamp‚1.784504333148254e+094íÄÊÀÄ & lr-AdamW/pg1‚0.0001072530222049362 & lr-AdamW/pg2‚0.0001072530222049362 trainer/global_step‚6049 $ _timestamp‚1.784504333148254e+09 _runtime‚ 2487.21141487 _step‚487线vZ2Á- _wandbruntime‚2487  _runtime‚2487áÁ>YØ'ÂÒO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 trainer/global_step‚6049 _runtime‚2487.214603048 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61285074 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚232884585145344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45958313 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61274931 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42113363 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚237134249793536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45958315 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 _step‚488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚237134249793536 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚226823946939392 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61285074 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4249681687552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26615247 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4249769709568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚232884585145344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚226824152256512 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45958313 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚232884568105984 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚237134354854912 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚237134354854912 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚222450199557120 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4373952699392 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4236397878638 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚222450174653440 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4373772285952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7845043331491792e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚229864699744176 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4249681687552 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 trainer/global_step‚6049 _runtime‚2487.214603048 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61285074 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚232884585145344 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45958313 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61274931 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42113363 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚237134249793536 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45958315 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 _step‚488 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚237134249793536 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚226823946939392 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61285074 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3è&&Ý…2Ä- _wandbruntime‚2487  _runtime‚2487äÞµÇá&ÅÛM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4374471198208 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 $ _timestamp‚1.784504333413117e+09 _runtime‚2487.476418024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15500538 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107260622 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚232921951512576 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4237073830082 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61294986 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61284829 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45965636 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61284829 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚237172294270464 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 107250463 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42120314 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229901571811936 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26619534 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚237172399339008 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚222485869810688 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4250359797248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107250463 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4250447826432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107260622 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26619537 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15500777 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6049 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229901554772576 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45965634 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚237172399339008 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42120072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚234138628602658 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26619534 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚237172399339008 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚222485869810688 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4250359797248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107250463 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4250447826432 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 107260622 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26619537 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15500777 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6049 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚229901554772576 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45965634 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚237172399339008 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42120072 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚234138628602658 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 107260622 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚232921951512576 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4237073830082 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚61294986 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61284829 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482éRQDÏ2Ç- _wandbruntime‚2487  _runtime‚2487ZŠXÈ /loss/FRt2/assignment_loss‚1.052527710090984 2loss/SRqqt1/assignment_loss‚0.7372856049981872 /loss/FRt1/detection_loss‚0.5721008481830321 trainer/global_step‚6049 % _timestamp‚1.7845043334140613e+09 2loss/FBt2/assignment_loss‚0.007663899163863686 1loss/SRqqt1/detection_loss‚0.9649381326906609 /loss/FBt2/detection_loss‚0.8652856234004911 %loss/total_loss‚11.08107705415998 epoch‚7 0loss/FRt1/assignment_loss‚2.5555774496760564 2loss/FBt1/assignment_loss‚0.035624358872523236 1loss/SRqqt2/detection_loss‚0.6285891737090423 /loss/FBt1/detection_loss‚1.6246054914971704 _runtime‚ 2487.48057312 _step‚490 2loss/SRqqt2/assignment_loss‚0.6644199769319837 /loss/FRt2/detection_loss‚1.3724587849459864µÙô Ɇ epoch‚7 0loss/FRt1/assignment_loss‚2.5555774496760564 2loss/FBt1/assignment_loss‚0.035624358872523236 1loss/SRqqt2/detection_loss‚0.6285891737090423 /loss/FBt1/detection_loss‚1.6246054914971704 _runtime‚ 2487.48057312 _step‚490 2loss/SRqqt2/assignment_loss‚0.6644199769319837 /loss/FRt2/detection_loss‚1.3724587849459864 /loss/FRt2/assignment_loss‚1.052527710090984 2loss/SRqqt1/assignment_loss‚0.7372856049981872 /loss/FRt1/detection_loss‚0.5721008481830321 trainer/global_step‚6049 % _timestamp‚1.7845043334140613e+09 2loss/FBt2/assignment_loss‚0.007663899163863686 1loss/SRqqt1/detection_loss‚0.9649381326906609 /loss/FBt2/detection_loss‚0.8652856234004911 %loss/total_loss‚11.08107705415998êýâeÊ‚(:Ú ÀõÒÅý›À disk.dm-1.in‚0 disk.nvme0c0n1.in‚91.03125! disk.nvme1c1n1.in‚ 10.48828125' proc.memory.availableMB‚ 193.4921875% disk./.usageGB‚1164.0675468444824 disk.rbd0.in‚0( disk.nvme0c0n1.out‚14423.57275390625 disk.nvme1n1.in‚ 10.48828125 cpu‚26.527519816160112 network.sent‚ 7.949424e+06 network.recv‚722793 disk.dm-0.in‚ 101.51953125" disk.dm-1.out‚1825.05322265625 disk.rbd0.out‚0$ disk.nvme1c1n1.out‚ 3608.77734375 proc.memory.rssMB‚ 2869.28125 proc.cpu.threads‚17 disk.nvme0n1.in‚91.03125& disk.nvme0n1.out‚14423.57275390625" disk.nvme1n1.out‚ 3608.77734375 disk.md0.in‚0 disk.md0.out‚ 73.234375$ memory_percent‚98.81901741027832) proc.memory.percent‚17.51270294189453) disk./.usagePercent‚75.82267036928725" disk.dm-0.out‚16051.0126953125Q bé›Ë‚(: ÀõÒ“ãÃÀ gpu.0.gpu‚62.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚204.173' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚68.05766666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>3==´Ì‚(:á žÀõÒÛãí¿& disk.nvme0n1.out‚14465.30810546875 disk.md0.in‚0# disk.dm-0.out‚16092.85693359375 disk.dm-1.in‚0 network.recv‚727284 disk.rbd0.out‚0 disk.nvme0c0n1.in‚91.03125! disk.nvme1c1n1.in‚ 10.48828125 disk.nvme1n1.in‚ 10.48828125 disk.md0.out‚ 73.66015625" disk.dm-1.out‚1825.12744140625) proc.memory.percent‚17.51270294189453 disk.nvme0n1.in‚91.03125% disk.nw?ä߸vme1n1.out‚3609.89892578125$ memory_percent‚99.21512603759766% proc.memory.availableMB‚ 128.59375 proc.memory.rssMB‚ 2869.28125 proc.cpu.threads‚17% disk./.usageGB‚1164.0397987365723( disk.nvme0c0n1.out‚14465.30810546875' disk.nvme1c1n1.out‚3609.89892578125 disk.dm-0.in‚ 101.51953125 cpu‚26.523691947668752 network.sent‚ 7.998844e+06) disk./.usagePercent‚75.82086296932566 disk.rbd0.in‚0M샚͂(: žÀõÒž’Û¿ gpu.0.gpu‚73.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚190.52' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚63.50666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•Z2Î- _wandbruntime‚2506  _runtime‚2506I¬yÉÏà _step‚491 ' lr-AdamW/pg1‚0.00010716110881941146 ' lr-AdamW/pg2‚0.00010716110881941146 trainer/global_step‚6099 % _timestamp‚1.7845043525840669e+09 _runtime‚2506.648593746©1ÜÐÎÐÈ trainer/global_step‚6099 % _timestamp‚1.7845043525840669e+09 _runtime‚2506.648593746 _step‚491 ' lr-AdamW/pg1‚0.00010716110881941146 ' lr-AdamW/pg2‚0.00010716110881941146ëžåh+2Ñ- _wandbruntime‚2506  _runtime‚2506œÒ²;Ø'ÒÒO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 108106191 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚234752903506944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46324365 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46324363 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61771683 % _timestamp‚1.7845043525847306e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26829747 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚235978513446862 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61771683 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4283619429376 _step‚492 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15621255 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224234957419520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 trainer/global_step‚6099 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4283707451392 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚224234932515840 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚231708303132176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46324365 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61781826 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚235978616634696 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61781826 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42451259 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚239036505896960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚234752903506944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚234752886467584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15621512 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228643702244864 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4408769729024 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4270313502520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚239036505896960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231708286092816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4283707451392 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46324363 _runtime‚2506.648648646 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚239036610958336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42450999 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4408952239616 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26829744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482ò ÍÝ'Ó×O LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4283707451392 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46324363 _runtime‚2506.648648646 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚239036610958336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚42450999 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4408952239616 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26829744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 108106191 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚234752903506944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46324365 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46324363 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61771683 % _timestamp‚1.7845043525847306e+09 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26829747 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚235978513446862 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61771683 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4283619429376 _step‚492 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15621255 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚224234957419520 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 trainer/global_step‚6099 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4283707451392 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚224234932515840 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚231708303132176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46324365 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61781826 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚235978616634696 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61781826 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42451259 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚239036505896960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚234752903506944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚234752886467584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15621512 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228643702244864 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4408769729024 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4270313502520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚239036505896960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚231708286092816ì|Ãô2Ô- _wandbruntime‚2506  _runtime‚2506.šâ&ÕÜM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚224270652576768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚234790252834816 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 trainer/global_step‚6099 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46331684 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚234790269874176 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4284385568256 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚239074655442432 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46331686 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46331684 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4409653008384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚236016061466066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108113265 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108123424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚224270627673088 % _timestamp‚1.7845043528507807e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61781581 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚239074655442432 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚236016164654012 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚234790269874176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4284385568256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228680305585152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108123424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61791738 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46331686 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4270989454076 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26834037 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15623659 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108113265 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42457964 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4409466310656 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚231745175199936 _runtime‚2506.914043939 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4284297539072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚493 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4270903305490 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚239074550373888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42457693 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚228680093983744 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚234790252834816 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408ƒý^)ÖáM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46331686 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4270989454076 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26834037 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15623659 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.smažBÖJ#ll_pool.allocated‚152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108113265 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42457964 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4409466310656 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚231745175199936 _runtime‚2506.914043939 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4284297539072 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚493 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4270903305490 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚239074550373888 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42457693 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚228680093983744 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚234790252834816 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚224270652576768 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚234790252834816 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 trainer/global_step‚6099 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46331684 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚234790269874176 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4284385568256 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚239074655442432 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46331686 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46331684 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4409653008384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚236016061466066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108113265 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108123424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚224270627673088 % _timestamp‚1.7845043528507807e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚61781581 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚239074655442432 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚236016164654012 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚234790269874176 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4284385568256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228680305585152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108123424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61791738ív Z¾2×- _wandbruntime‚2506  _runtime‚2506©ìFúØÿ /loss/FRt1/assignment_loss‚2.512445684987033 1loss/FBt1/assignment_loss‚0.08736513020322371 1loss/FBt2/assignment_loss‚0.05126883139585371 1loss/SRqqt1/detection_loss‚1.0454001152564003 /loss/FBt2/detection_loss‚0.5337941520491667 0loss/FRt2/assignment_loss‚1.1474126229683559 .loss/FRt2/detection_loss‚1.543675070950052 1loss/SRqqt2/detection_loss‚0.7036532158213399 &loss/total_loss‚10.717126833533587 epoch‚7 /loss/FRt1/detection_loss‚0.5916651732420637 % _timestamp‚1.7845043528517532e+09 _runtime‚ 2506.91456912 2loss/SRqqt1/assignment_loss‚0.8218149930117247 2loss/SRqqt2/assignment_loss‚0.6521786291890709 /loss/FBt1/detection_loss‚1.0264532144593028 trainer/global_step‚6099 _step‚494¶N,Á Ù„ /loss/FRt1/assignment_loss‚2.512445684987033 1loss/FBt1/assignment_loss‚0.08736513020322371 1loss/FBt2/assignment_loss‚0.05126883139585371 1loss/SRqqt1/detection_loss‚1.0454001152564003 /loss/FBt2/detection_loss‚0.5337941520491667 0loss/FRt2/assignment_loss‚1.1474126229683559 .loss/FRt2/detection_loss‚1.543675070950052 1loss/SRqqt2/detection_loss‚0.7036532158213399 &loss/total_loss‚10.717126833533587 epoch‚7 /loss/FRt1/detection_loss‚0.5916651732420637 % _timestamp‚1.7845043528517532e+09 _runtime‚ 2506.91456912 2loss/SRqqt1/assignment_loss‚0.8218149930117247 2loss/SRqqt2/assignment_loss‚0.6521786291890709 /loss/FBt1/detection_loss‚1.0264532144593028 trainer/global_step‚6099 _step‚494î0Éî]aÚ‚(:Ö ­ÀõÒ±á’À' disk.nvme0c0n1.out‚14466.8505859375 disk.nvme0n1.in‚91.03125 disk.nvme1n1.in‚ 10.48828125 disk.md0.out‚ 74.17578125 disk.dm-0.in‚ 101.51953125! disk.dm-0.out‚16094.126953125& disk.nvme1c1n1.out‚3778.2314453125$ memory_percent‚99.21400547027588( proc.memory.availableMB‚ 128.77734375 proc.memory.rssMB‚ 2869.28125 network.sent‚ 8.017629e+06 disk.dm-1.in‚0 disk.nvme0c0n1.in‚91.03125! disk.nvme1c1n1.in‚ 10.48828125) proc.memory.percent‚17.51270294189453 cpu‚26.519993719505973) disk./.usagePercent‚75.82084259445669$ disk.nvme1n1.out‚3778.2314453125% disk.nvme0n1.out‚14466.8505859375 proc.cpu.threads‚17 network.recv‚730667% disk./.usageGB‚1164.0394859313965 disk.md0.in‚0 disk.dm-1.out‚1992.625 disk.rbd0.in‚0 disk.rbd0.out‚0$€'›Û‚(: ­ÀõÒ™ªÙ¿ gpu.0.gpu‚60.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚201.281' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.09366666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãrÖƒ2Ü- _wandbruntime‚2526  _runtime‚2526fdï©ÈÝ $ _timestamp‚1.784504372012192e+09 _runtime‚2526.076013928 _step‚495 ' lr-AdamW/pg1‚0.00010706837700209686 ' lr-AdamW/pg2‚0.00010706837700209686 trainer/global_step‚6149“qg’ÍÞÇ ' lr-AdamW/pg1‚0.00010706837700209686 ' lr-AdamW/pg2‚0.00010706837700209686 trainer/global_step‚6149 $ _timestamp‚1.784504372012192e+09 _runtime‚2526.076013928 _step‚495ï—OÉ2ß- _wandbruntime‚2526  _runtime‚2526Ø'àÒO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚240938762000384 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚42789004 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4443752006144 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 _runtime‚2526.076071288 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚240938867061760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15744757 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚230463442384384 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62268435 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4317645193216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46690415 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚226019690378240 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4304056829454 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚236621204829184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62268435 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚230463651895808 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚237856049498104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4317645193216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚236621221868544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62278578 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚240938867061760 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚240938762000384 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚236621204829184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚233551889480816 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4304142977928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27044247 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62278578 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 _runtime‚2526.076071288 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚240938867061760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15744757 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚230463442384384 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62268435 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4317645193216 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46690415 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚226019690378240 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4304056829454 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚236621204829184 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62268435 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚230463651895808 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚237856049498104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4317645193216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚236621221868544 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62278578 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚240938867061760 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚240938762000384 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚236621204829184 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚233551889480816 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4304142977928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27044247 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62278578 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108986226 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46697734 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42795517 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚237893597517420 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62288490 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108976067 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚222 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚230500044413952 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4318235280896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚236658571196416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚237893494329474 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15747205 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4304818929484 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚240976806477312 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108986226 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108976067 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4444633974784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4444453568512 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚233588778587936 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 trainer/global_step‚6149 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚240976911545856 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42795742 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚236658588235776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚240976911545856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 _runtime‚2526.339359105 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46697736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚236658571196416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248#€¡+ç&äáM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108986226 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 108976067 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4444633974784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4444453568512 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚233588778587936 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 trainer/global_step‚6149 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚240976911545856 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚42795742 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚236658588235776 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚240976911545856 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 _runtime‚2526.339359105 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46697736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚236658571196416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚230499839104000 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 % _timestamp‚1.7845043722760592e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62278333 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108986226 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46697734 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42795517 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚237893597517420 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62288490 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 108976067 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚222 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚230500044413952 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4318235280896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚236658571196416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚237893494329474 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15747205 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4304818929484 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚240976806477312ñ(RØß2å- _wandbruntime‚2526  _runtime‚2526)c›æ€ _step‚498 /loss/FBt1/detection_loss‚0.6694044535016191 /loss/FRt1/assignment_loss‚2.633736627673671 2loss/SRqqt2/assignment_loss‚0.6684307923176165 2loss/FBt2/assignment_loss‚0.018472049046014332 _runtime‚2526.339903234 trainer/global_step‚6149 % _timestamp‚1.7845043722770395e+09 /loss/FBt2/detection_loss‚0.7329803341429453 &loss/total_loss‚10.895599869721993 epoch‚7 2loss/SRqqt1/assignment_loss‚0.8864197668715798 0loss/FBt1/assignment_loss‚0.1440528294973037 /loss/FRt1/detection_loss‚0.5883261344553163 .loss/FRt2/detection_loss‚1.445411260270547 1loss/SRqqt1/detection_loss‚1.2506722870234657 0loss/FRt2/assignment_loss‚1.1424347605024066 1loss/SRqqt2/detection_loss‚0.7152585744195068ðêX" ç… /loss/FRt1/assignment_loss‚2.633736627673671 2loss/SRqqt2/assignment_loss‚0.6684307923176165 2loss/FBt2/assignment_loss‚0.018472049046014332 _runtime‚2526.339903234 trainer/global_step‚6149 % _timestamp‚1.7845043722770395e+09 /loss/FBt2/detection_loss‚0.7329803341429453 &loss/total_loss‚10.895599869721993 epoch‚7 2loss/SRqqt1/assignment_loss‚0.8864197668715798 0loss/FBt1/assignment_loss‚0.1440528294973037 /loss/FRt1/detection_loss‚0.5883261344553163 .loss/FRt2/detection_loss‚1.445411260270547 1loss/SRqqt1/detection_loss‚1.2506722870234657 0loss/FRt2/assignment_loss‚1.1424347605024066 1loss/SRqqt2/detection_loss‚0.7152585744195068 _step‚498 /loss/FBt1/detection_loss‚0.6694044535016191òÜgè‚(:Ü ¼ÀõÒ‘«Ì¿ network.recv‚735245) disk./.usagePercent‚75.82083961276855& disk.nvme0n1.out‚14469.05810546875 disk.dm-0.in‚ 101.51953125 disk.rbd0.out‚0 proc.memory.rssMB‚ 2869.28125 cpu‚26.516455418135678 network.sent‚ 8.065924e+06 disk.nvme0n1.in‚91.03125! disk.nvme1c1n1.in‚ 10.48828125 disk.nvme1n1.in‚ 10.48828125 disk.nvme0c0n1.in‚91.03125( disk.nvme0c0n1.out‚14469.05810546875) proc.memory.percent‚17.51270294189453 proc.cpu.threads‚17% disk./.usageGB‚1164.0394401550293# disk.nvme1n1.out‚3859.912109375 disk.md0.out‚ 74.63671875 disk.dm-1.in‚0' proc.memory.availableMB‚ 128.7421875% disk.nvme1c1n1.out‚3859.912109375 disk.md0.in‚0# disk.dm-0.out‚16095.97802734375 disk.dm-1.out‚ 2073.66796875 disk.rbd0.in‚0$ memory_percent‚99.21422004699707°d_Åé‚(:… ¼ÀõÒ À gpu.0.gpu‚58.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚194.091' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚64.697 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ý@’ò2ê- _wandbruntime‚2545  _runtime‚2545¨Ÿ Éëà % _timestamp‚1.7845043911120105e+09 _runtime‚2545.175166165 _step‚499 ' lr-AdamW/pg1‚0.00010697482823828068 ' lr-AdamW/pg2‚0.00010697482823828068 trainer/global_step‚6199rç£ÈÎìÈ _runtime‚2545.175166165 _step‚499 ' lr-AdamW/pg1‚0.00010697482823828068 ' lr-AdamW/pg2‚0.00010697482823828068 trainer/global_step‚6199 % _timestamp‚1.7845043911120105e+09óøè2í- _wandbruntime‚2545  _runtime‚2545C?˜¸å#îÒO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62765187 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚238489540230144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚238489540230144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚238489523190784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚235395509908176 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109821650 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47056465 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4337886304862 _step‚500 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚239733379173678 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242841018103808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4351582935040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109831795 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43126651 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚242841123165184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62775330 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15867904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚227804448240640 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚239733482361512 trainer/global_step‚6199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4351494913024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚227804473144320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47056463 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚238489523190784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚242841123165184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4351582935040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚2545.176095507 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚232283122461184 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62775330 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚242841018103808 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43126406 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27258747 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4478674220544 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4337972453336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62765187 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7845043911130238e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27258744 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_starŠ±Ó!ót/segment.large_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚235395492868816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4351494913024 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚232283329875456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47056465 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0ÔþeÝ'ï×O \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚235395492868816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4351494913024 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚232283329875456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47056465 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚62765187 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚238489540230144 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚238489540230144 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚238489523190784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚235395509908176 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109821650 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47056465 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4337886304862 _step‚500 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚239733379173678 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242841018103808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4351582935040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 109831795 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚43126651 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚242841123165184 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62775330 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15867904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚227804448240640 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚239733482361512 trainer/global_step‚6199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4351494913024 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚227804473144320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47056463 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚238489523190784 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚242841123165184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4351582935040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚2545.176095507 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚232283122461184 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚62775330 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚242841018103808 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43126406 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27258747 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4478674220544 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4337972453336 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62765187 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7845043911130238e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27258744 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ôù© 2ð- _wandbruntime‚2545  _runtime‚2545Ääs{â&ñÜM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62785242 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚239771030380828 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15870181 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚239770927192882 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43133215 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109849028 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27263034 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚227840168301568 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43133480 _runtime‚2545.749922689 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109849028 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4338562256306 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚242879167649280 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15870443 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚235432364936576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4352173022720 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47063784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚238526889558016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47063786 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62785242 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4338648404892 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 trainer/global_step‚6199 _step‚501 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47063786 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62775085 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚238526906597376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27263037 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚242879062580736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4479555992576 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4352173022720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4352261051904 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232319724294144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4479367197696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109838869 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚227840143397888 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47063784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚238526889558016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845043916865902e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚242879167649280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚235432381975936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚232319510595584 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ÊGäç&òáM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4352261051904 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232319724294144 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4479367197696 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109838869 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚227840143397888 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47063784 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚238526889558016 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845043916865902e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚242879167649280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚235432381975936 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚232319510595584 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚62785242 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚239771030380828 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15870181 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚239770927192882 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43133215 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109849028 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27263034 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚227840168301568 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43133480 _runtime‚2545.749922689 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 109849028 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4338562256306 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚242879167649280 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15870443 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚235432364936576 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4352173022720 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47063784 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚238526889558016 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47063786 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚62785242 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4338648404892 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 trainer/global_step‚6199 _step‚501 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47063786 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚62775085 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚238526906597376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27263037 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚242879062580736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4479555992576 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4352173022720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0õÞ’D2ó- _wandbruntime‚2545  _runtime‚2545J· ô„ .loss/FRt1/detection_loss‚0.555251107479868 &loss/total_loss‚10.134932815730775 epoch‚7 % _timestamp‚1.7845043916876452e+09 0loss/FRt2/assignment_loss‚0.9907439489856021 2loss/FBt1/assignment_loss‚0.030495771612564226 /loss/FBt2/detection_loss‚0.5662931489701993 _step‚502 0loss/FRt1/assignment_loss‚2.4824984065123967 2loss/SRqqt2/assignment_loss‚0.6844000243229992 /loss/FRt2/detection_loss‚1.3686181916588676 trainer/global_step‚6199 2loss/SRqqt1/assignment_loss‚0.6950140953791348 3loss/FBt2/assignment_loss‚0.0007552057853078361 1loss/SRqqt1/detection_loss‚1.0838685131624994 1loss/SRqqt2/detection_loss‚0.7677516899847303 /loss/FBt1/detection_loss‚0.9092427118766057 _runtime‚2545.750520381‘ÊKòÁõ‰ 3loss/FBt2/assignment_loss‚0.0007552057853078361 1loss/SRqqt1/detection_loss‚1.0838685131624994 1loss/SRqqt2/detection_loss‚0.7677516899847303 /loss/FBt1/detection_loss‚0.9092427118766057 _runtime‚2545.750520381 .loss/FRt1/detection_loss‚0.555251107479868 &loss/total_loss‚10.134932815730775 epoch‚7 % _timestamp‚1.7845043916876452e+09 0loss/FRt2/assignment_loss‚0.9907439489856021 2loss/FBt1/assignment_loss‚0.030495771612564226 /loss/FBt2/detection_loss‚0.5662931489701993 _step‚502 0loss/FRt1/assignment_loss‚2.4824984065123967 2loss/SRqqt2/assignment_loss‚0.6844000243229992 /loss/FRt2/detection_loss‚1.3686181916588676 train å«~Ner/global_step‚6199 2loss/SRqqt1/assignment_loss‚0.6950140953791348öVt1`oö‚(:ä ËÀõÒ›ó©À" disk.dm-1.out‚2076.45166015625( disk.nvme0c0n1.out‚14469.61181640625 disk.dm-0.in‚ 101.51953125( proc.memory.availableMB‚ 129.14453125 proc.cpu.threads‚17 disk.nvme0c0n1.in‚91.03125' disk.nvme1c1n1.out‚3862.90576171875 disk.nvme1n1.in‚ 10.48828125% disk.nvme1n1.out‚3862.90576171875 disk.md0.in‚0# disk.dm-0.out‚16096.42333984375 network.recv‚739480) disk./.usagePercent‚75.73040575686854 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚14469.61181640625! disk.nvme1c1n1.in‚ 10.48828125 disk.md0.out‚ 74.77734375$ memory_percent‚99.21176433563232 network.sent‚ 8.140965e+06% disk./.usageGB‚1162.6510543823242 disk.dm-1.in‚0 disk.nvme0n1.in‚91.03125 proc.memory.rssMB‚ 2869.28125) proc.memory.percent‚17.51270294189453 cpu‚26.512833261995482áY®w›÷‚(: ËÀõÒÄëÕÀ gpu.0.gpu‚51.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚160.322' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚53.44066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ȭ~k2ø- _wandbruntime‚2564  _runtime‚2564hõ`jÉùà _runtime‚2564.600511084 _step‚503 ' lr-AdamW/pg1‚0.00010688046402633624 ' lr-AdamW/pg2‚0.00010688046402633624 trainer/global_step‚6249 % _timestamp‚1.7845044105373514e+09¨o!PÎúÈ ' lr-AdamW/pg1‚0.00010688046402633624 ' lr-AdamW/pg2‚0.00010688046402633624 trainer/global_step‚6249 % _timestamp‚1.7845044105373514e+09 _runtime‚2564.600511084 _step‚503÷¼Ûç!2û- _wandbruntime‚2564  _runtime‚2564ïÆ3º×'üÑO JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4385520676864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63272082 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚234102798151168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63272082 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚240357858591744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27473247 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4513592048128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 110684452 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚240357858591744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244743274207232 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚244743274207232 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚241610812037086 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27473244 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚241610915224920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4385520676864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚237239113296176 _runtime‚2564.604643302 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15992083 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚240357841552384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4371801928744 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚244743379268608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 110694597 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚234103007662592 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4371715780270 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚244743379268608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63261939 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚229589206103040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚237239096256816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47422515 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.784504410538294e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4385432654848 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374•¶@Ü'ýÖO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47422515 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4385520676864 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63272082 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚234102798151168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63272082 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚240357858591744 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27473247 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4513592048128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 110684452 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚240357858591744 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚244743274207232 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚244743274207232 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚241610812037086 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27473244 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚241610915224920 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4385520676864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚237239113296176 _runtime‚2564.604643302 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15992083 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚240357841552384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4371801928744 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚244743379268608 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 110694597 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚234103007662592 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4371715780270 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚244743379268608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63261939 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚229589206103040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚237239096256816 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚47422515 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.784504410538294e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4385432654848 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43472060 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27477534 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47429836 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63271837 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚244781318684160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4372391731714 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15994526 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110711830 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4386198793728 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4514471522304 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚6249 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚229624901260288 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚237275985363936 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4372477880300 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 $ _timestamp‚1.784504410802294e+09 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚244781423752704 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _runtime‚2564.865629665 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 110711830 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚244781423752704 _step‚505 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15994775 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚237275968324576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚240395224958976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4386110764544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4514289018880 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47429834 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63281994 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚240395207919616 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4386110764544 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63271837 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚234139190279168¤ TÖ“€àM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15994775 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚237275968324576 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocatioPO¼S%ns.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚240395224958976 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4386110764544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4514289018880 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚47429834 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63281994 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚240395207919616 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4386110764544 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63271837 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚234139190279168 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚240395207919616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4386198793728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63281994 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43472060 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27477534 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47429836 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63271837 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚244781318684160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4372391731714 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15994526 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 110711830 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4386198793728 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4514471522304 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚6249 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚229624901260288 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚237275985363936 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4372477880300 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 $ _timestamp‚1.784504410802294e+09 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚244781423752704 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _runtime‚2564.865629665 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 110711830 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚244781423752704 _step‚505 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544ùè­2- _wandbruntime‚2564  _runtime‚2564ÆÖ%9‚ÿ /loss/FRt1/assignment_loss‚2.695559577851356 2loss/SRqqt1/assignment_loss‚0.8630380029385883 1loss/FBt2/assignment_loss‚0.08309555196464001 &loss/total_loss‚10.827557327504625 epoch‚7 _runtime‚2564.869578322 2loss/SRqqt2/assignment_loss‚0.6431843265708137 /loss/FRt1/detection_loss‚0.5016602458815947 /loss/FRt2/detection_loss‚1.4262844738533174 1loss/SRqqt1/detection_loss‚1.2214288884101228 1loss/SRqqt2/detection_loss‚0.6839698313519226 _step‚506 0loss/FRt2/assignment_loss‚1.0587867671229139 1loss/FBt1/assignment_loss‚0.07711310080058016 /loss/FBt2/detection_loss‚0.6616241610656568 % _timestamp‚1.7845044108033721e+09 -loss/FBt1/detection_loss‚0.91181239969312 trainer/global_step‚6249Pô2S ƒ„ 2loss/SRqqt2/assignment_loss‚0.6431843265708137 /loss/FRt1/detection_loss‚0.5016602458815947 /loss/FRt2/detection_loss‚1.4262844738533174 1loss/SRqqt1/detection_loss‚1.2214288884101228 1loss/SRqqt2/detection_loss‚0.6839698313519226 _step‚506 0loss/FRt2/assignment_loss‚1.0587867671229139 1loss/FBt1/assignment_loss‚0.07711310080058016 /loss/FBt2/detection_loss‚0.6616241610656568 % _timestamp‚1.7845044108033721e+09 -loss/FBt1/detection_loss‚0.91181239969312 trainer/global_step‚6249 /loss/FRt1/assignment_loss‚2.695559577851356 2loss/SRqqt1/assignment_loss‚0.8630380029385883 1loss/FBt2/assignment_loss‚0.08309555196464001 &loss/total_loss‚10.827557327504625 epoch‚7 _runtime‚2564.869578322ú¬Údd„‚(:Ù ÚÀõÒÞÎï¿ disk.rbd0.out‚0! disk.nvme1c1n1.in‚ 10.48828125$ memory_percent‚99.21324253082275) proc.memory.percent‚17.51270294189453 network.recv‚743891% disk./.usageGB‚1162.8888702392578 disk.nvme0n1.in‚91.03125% disk.nvme0n1.out‚14662.3017578125' disk.nvme1c1n1.out‚3916.16357421875& disk./.usagePercent‚75.74589612371 disk.md0.in‚0 disk.md0.out‚ 75.4921875 disk.dm-0.in‚ 101.51953125" disk.dm-1.out‚2076.65869140625 proc.memory.rssMB‚ 2869.28125 proc.cpu.threads‚17 network.sent‚ 8.190478e+06 disk.nvme1n1.in‚ 10.48828125 disk.nvme0c0n1.in‚91.03125' disk.nvme0c0n1.out‚14662.3017578125( proc.memory.availableMB‚ 128.90234375 cpu‚26.509375735386577% disk.nvme1n1.out‚3916.16357421875 disk.dm-0.out‚ 16340.640625 disk.dm-1.in‚0 disk.rbd0.in‚0½‡Î‘…‚(:† ÚÀõÒ÷¥ÍÀ gpu.0.gpu‚88.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚134.388' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚44.796 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ë»ç1n†‚(:ã éÀõÒÓµƒÀ$ memory_percent‚99.21200275421143 cpu‚26.505848749252845 proc.cpu.threads‚17 network.sent‚ 8.254654e+06 disk.dm-0.in‚ 101.51953125 disk.rbd0.in‚0( proc.memory.availableMB‚ 129.10546875) proc.memory.percent‚17.51270294189453) disk./.usagePercent‚75.74589165117779! disk.nvme1c1n1.in‚ 10.48828125' disk.nvme1c1n1.out‚3918.68017578125% disk.nvme1n1.out‚3918.68017578125 disk.md0.out‚ 76.10546875# disk.dm-0.out‚16344.48095703125" disk.dm-1.out‚2078.42236328125 proc.memory.rssMB‚ 2869.28125 network.recv‚748231 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚91.03125( disk.nvme0c0n1.out‚14666.69970703125& disk.nvme0n1.out‚14666.69970703125$ disk./.usageGB‚1162.888801574707 disk.nvme1n1.in‚ 10.48828125 disk.md0.in‚0 disk.nvme0n1.in‚91.03125ÑÖ†vœ‡‚(:‘ éÀõÒƯÀ gpu.0.gpu‚84.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚194.646' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚64.88199999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ûx^f2ˆ- _wandbruntime‚2584  _runtime‚2584–{Mȉ ' lr-AdamW/pg1‚0.00010678528587769798 ' lr-AdamW/pg2‚0.00010678528587769798 trainer/global_step‚6299 $ _timestamp‚1.784504430033443e+09 _runtime‚2584.096582954 _step‚507!vcÍŠÇ $ _timestamp‚1.784504430033443e+09 _runtime‚2584.096582954 _step‚507 ' lr-AdamW/pg1‚0.00010678528587769798 ' lr-AdamW/pg2‚0.00010678528587769798 trainer/global_step‚6299û¯Ç,2‹- _wandbruntime‚2584  _runtime‚2584óÓØ'ŒÒO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚47788563 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚231373963965440 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4405631404152 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚246645635372032 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4548537047552 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4419458418688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63768834 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27687747 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚235922501012992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4405545255678 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63758691 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚239082716684176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚239082699644816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 111547254 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246645530310656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚231373988869120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚6299 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚246645530310656 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 _step‚508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63758691 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚243488244900494 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4419370396672 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4419370396672 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 % _timestamp‚1.7845044300343826e+09 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47788565 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚242226176953344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27687744 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 % _timestamp‚1.7845044300343826e+09 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚47788565 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚242226176953344 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27687744 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚235922501012992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4405545255678 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63758691 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚239082716684176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚239082699644816 F@DeviceStatsMARõý£onitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 111547254 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246645530310656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚231373988869120 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚6299 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚246645530310656 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 _step‚508 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63758691 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚243488244900494 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104ü3–ló2Ž- _wandbruntime‚2584  _runtime‚2584Z¿,yâ&ÜM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27692034 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111574632 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235958898387968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4406307355708 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4420048506368 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4420136535552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚246683679856128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47795886 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63778746 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43810729 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚242263543320576 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111574632 trainer/global_step‚6299 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚242263526281216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚231409659122688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚242263526281216 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47795886 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚242263543320576 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 % _timestamp‚1.7845044302985175e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27692037 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4549419671552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 _runtime‚2584.361834366 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4549239265280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111564473 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63768589 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47795884 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152º›Þç&áM E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚242263526281216 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚47795886 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚242263543320576 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 % _timestamp‚1.7845044302985175e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27692037 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4549419671552 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 _runtime‚2584.361834366 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4549239265280 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111564473 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63768589 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚47795884 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27692034 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111574632 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235958898387968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4406307355708 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4420048506368 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4420136535552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚246683679856128 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚47795886 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63778746 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43810729 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚242263543320576 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111574632 trainer/global_step‚6299 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚242263526281216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚231409659122688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚244094495314944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44143064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚237742468075008 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚245365780951736 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚248547786414080 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48154615 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64255443 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4453308138496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 trainer/global_step‚6349 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4453308138496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 112420201 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16240820 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚248547891475456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27902247 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240926320072176 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚233158721827840 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4439374731086 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44143310 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27902244 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4453396160512 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚237742260660736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚233158746731520 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48154615 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚512 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64265586 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48154613 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4583538832896 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4439460879560 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4453396160512 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚248547891475456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.“•a on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚245365677763902 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 112410056 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚248547786414080 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240926303032816 % _timestamp‚1.7845044495050104e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚244094495314944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48154613 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚512 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64265586 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48154613 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4583538832896 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4439460879560 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4453396160512 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚248547891475456 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚245365677763902 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 112410056 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚248547786414080 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240926303032816 % _timestamp‚1.7845044495050104e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚244094495314944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48154613 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚244094495314944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44143064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚237742468075008 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚245365780951736 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚248547786414080 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48154615 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64255443 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4453308138496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 trainer/global_step‚6349 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4453308138496 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 112420201 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16240820 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚248547891475456 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27902247 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚240926320072176 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚233158721827840 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4439374731086 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44143310 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27902244 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482€~±'2œ- _wandbruntime‚2603  _runtime‚2603ûûâ&ÜM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44150010 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112427275 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚244131861682176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4584423554048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚248585935959552 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 trainer/global_step‚6349 _step‚513 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4453986248192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚244131861682176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44149750 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64265341 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48161936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112437434 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48161936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64275498 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4584232662016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16243216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚245403328971052 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚240963175100576 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4453986248192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚2603.832587329 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64275498 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48161934 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚237778649647104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16243473 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64265341 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚233194416985088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27906534 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚244131844642816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚237778865442816 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚245403225783106 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4440050682530 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚248585935959552 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 112427275 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 112437434 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48161934 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4454074277376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚233194441888768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27906537 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4440136831116 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 112437434 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48161934 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4454074277376 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚233194441888768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27906537 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4440136831116 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112427275 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚244131861682176 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4584423554048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚248585935959552 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 trainer/global_step‚6349 _step‚513 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4453986248192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚244131861682176 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44149750 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64265341 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48161936 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112437434 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48161936 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚64275498 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4584232662016 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16243216 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚245403328971052 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚240963175100576 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4453986248192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚2603.832587329 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64275498 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48161934 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚237778649647104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16243473 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64265341 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚257 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚233194416985088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27906534 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚244131844642816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚237778865442816 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚245403225783106 D>DeviceStatsMonitor.on_train_baøïDtch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4440050682530 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ƾM2Ÿ- _wandbruntime‚2603  _runtime‚2603„gˆè ÿ 0loss/FRt2/assignment_loss‚1.0430246367773601 1loss/FBt2/assignment_loss‚0.05477285405725702 /loss/FBt2/detection_loss‚0.5696465503048873 epoch‚7 trainer/global_step‚6349 # _timestamp‚1.78450444977031e+09 _step‚514 /loss/FRt1/detection_loss‚0.5805685984540188 .loss/FRt2/detection_loss‚1.443868478607788 &loss/total_loss‚10.336864441184574 /loss/FRt1/assignment_loss‚2.595925710507943 2loss/SRqqt2/assignment_loss‚0.6418071429409139 1loss/SRqqt2/detection_loss‚0.6445313451332892 /loss/FBt1/detection_loss‚0.6644119535657492 2loss/SRqqt1/assignment_loss‚0.9361115243587712 2loss/FBt1/assignment_loss‚0.028085915823682098 1loss/SRqqt1/detection_loss‚1.1341097306529129 _runtime‚2603.833668532Û¿< ¡„ _runtime‚2603.833668532 0loss/FRt2/assignment_loss‚1.0430246367773601 1loss/FBt2/assignment_loss‚0.05477285405725702 /loss/FBt2/detection_loss‚0.5696465503048873 epoch‚7 trainer/global_step‚6349 # _timestamp‚1.78450444977031e+09 _step‚514 /loss/FRt1/detection_loss‚0.5805685984540188 .loss/FRt2/detection_loss‚1.443868478607788 &loss/total_loss‚10.336864441184574 /loss/FRt1/assignment_loss‚2.595925710507943 2loss/SRqqt2/assignment_loss‚0.6418071429409139 1loss/SRqqt2/detection_loss‚0.6445313451332892 /loss/FBt1/detection_loss‚0.6644119535657492 2loss/SRqqt1/assignment_loss‚0.9361115243587712 2loss/FBt1/assignment_loss‚0.028085915823682098 1loss/SRqqt1/detection_loss‚1.1341097306529129‚ù(rjm¢‚(:â ‡ÁõÒ¹¯›Á disk.rbd0.in‚0& disk.nvme0n1.out‚14681.35986328125 cpu‚26.503756109528744# disk.dm-0.out‚16358.61181640625" disk.dm-1.out‚2147.00830078125 disk.nvme0c0n1.in‚91.03125( disk.nvme0c0n1.out‚14681.35986328125' disk.nvme1c1n1.out‚3989.38134765625) proc.memory.percent‚17.51270294189453 disk.nvme1n1.in‚ 10.48828125% disk.nvme1n1.out‚3989.38134765625 disk.md0.in‚0! disk.nvme1c1n1.in‚ 10.48828125$ memory_percent‚99.21259880065918 proc.cpu.threads‚17) disk./.usagePercent‚75.74589960234617% disk./.usageGB‚1162.8889236450195 disk.md0.out‚ 77.3671875 disk.dm-0.in‚ 101.51953125 disk.rbd0.out‚0 disk.nvme0n1.in‚91.03125' proc.memory.availableMB‚ 129.0078125 proc.memory.rssMB‚ 2869.28125 network.sent‚ 8.319988e+06 network.recv‚755638 disk.dm-1.in‚0ÉŤš£‚(: ‡ÁõÒ±ÙÄÁ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.689' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.22966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y݃2¤- _wandbruntime‚2623  _runtime‚26233ŸZhɥà ' lr-AdamW/pg1‚0.00010659249388123723 ' lr-AdamW/pg2‚0.00010659249388123723 trainer/global_step‚6399 % _timestamp‚1.7845044695327783e+09 _runtime‚2623.595923477 _step‚515²C±rÎ¦È trainer/global_step‚6399 % _timestamp‚1.7845044695327783e+09 _runtime‚2623.595923477 _step‚515 ' lr-AdamW/pg1‚0.00010659249388123723 ' lr-AdamW/pg2‚0.00010659249388123723ƒ-ªH2§- _wandbruntime‚2623  _runtime‚26234Í”äØ'¨ÒO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16363945 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _runtime‚2623.596884659 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚250450147578880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64752195 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚6399 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4487333902336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4487245880320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚250450147578880 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16363686 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 % _timestamp‚1.7845044695337539e+09 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚245962796637184 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48520665 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4473204206494 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚247243213815144 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚250450042517504 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚242769923460176 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48520663 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64762338 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113283003 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚250450042517504 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44480692 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4618509771264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64752195 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44480430 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28116744 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4618696476160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64762338 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4473290354968 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113272858 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4487333902336 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚234943504593920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚245962796637184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚239561989461504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _step‚516 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚245962813676544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4487245880320 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚242769906420816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48520663ºÞ\lÝ'©×O G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚250450147578880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚64752195 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚6399 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4487333902336 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4487245880320 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚250450147578880 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16363686 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 % _timestamp‚1.7845044695337539e+09 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚245962796637184 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48520665 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4473204206494 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚247243213815144 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚250450042517504 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚242769923460176 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48520663 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚64762338 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 113283003 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚250450042517504 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44480692 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4618509771264 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚64752195 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚44480430 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28116744 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4618696476160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚64762338 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4473290354968 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 113272858 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4487333902336 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚234943504593920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚245962796637184 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚239561989461504 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _step‚516 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚245962813676544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4487245880320 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚242769906420816 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48520663 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16363945 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _runtime‚2623.596884659 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48527986 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4487923990016 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚250488086994432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113300236 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚250488192062976 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44487156 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4619393832960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4473966306524 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113300236 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚242806778488576 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚234979174847488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64772250 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚517 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚247280761834460 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48527984 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44487402 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16366122 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚234979199751168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚246000180043776 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚242806795527936 XFDeviceStatsMonitor.on_train_batch_end/requested_bytá7Ím‚es.small_pool.freed‚ 4473880157938 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚246000163004416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚247280658646514 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48527986 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚246000163004416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48527984 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4488012019200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28121037 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16366365 trainer/global_step‚6399 _runtime‚2623.862011573 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816!é£æ&¬àM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4619211329536 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4473966306524 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 113300236 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚242806778488576 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚234979174847488 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64772250 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚517 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚247280761834460 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48527984 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44487402 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16366122 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚234979199751168 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚246000180043776 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚242806795527936 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4473880157938 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚246000163004416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚247280658646514 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48527986 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚246000163004416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48527984 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4488012019200 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28121037 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16366365 trainer/global_step‚6399 _runtime‚2623.862011573 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64762093 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 $ _timestamp‚1.784504469798691e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚250488192062976 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4488012019200 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64762093 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28121034 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚246000180043776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚239598593584128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 113290077 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4487923990016 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48527986 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4487923990016 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚250488086994432 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 113300236 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚250488192062976 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44487156 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4619393832960 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 F&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07500000298023224 particle/accuracy_1_of_5‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.532256007194519 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01769634522497654 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13686801493167877 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15461890399456024 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6345404982566833 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_2_of_3‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8021367192268372 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6751890778541565 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1200888380408287 jet/accuracy_4_of_5‚0 _step‚519 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6885814666748047 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06150821968913078 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6132687330245972 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_1_of_1‚0.3018118441104889 7validation_average_jet_accuracy‚0.34753134846687317 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9808877110481262 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4678795635700226 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7428649663925171 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2777777910232544 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0005668934318237007 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01895691268146038 *jet/accuracy_2_of_2‚0.1366782784461975 particle/accuracy_2_of_4‚1 particle/accuracy_5_of_5‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7467973232269287 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20371100306510925 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6555129885673523 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.46174076199531555 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1701967418193817 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5174086689949036 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6261477470397949 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.035790491849184036 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15499350428581238 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.049154382199048996 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10304297506809235 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7356455326080322 particle/accuracy_3_of_4‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9642857313156128 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 'particle/f_score‚0.6392379999160767 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9940814971923828 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9642021656036377 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4388047158718109 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11562500149011612 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0714285746216774 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0714285746216774 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 *jet/accuracy_2_of_4‚0.7985166907310486 +particle/specificity‚0.9092022776603699 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_4_of_5‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15402621030807495 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2819328010082245 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0005668934318237007 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21989865601062775 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.830365777015686 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.830365777015686 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4359224736690521 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6215733289718628 ,jet/accuracy_4_of_4‚0.014682539738714695 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9988662004470825 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9953886866569519 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9953965544700623 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11940373480319977 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9060606956481934 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17389948666095734 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.049154382199048996 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.73295658826828 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16613391041755676 +jet/accuracy_3_of_4‚0.08360614627599716 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8241226673126221 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7400000095367432 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995591044425964 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.48890626430511475 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.27272728085517883 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.010032395832240582 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.14958932995796204 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1670842468738556 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.995820164680481 )jet/accuracy_1_of_2‚0.768553614616394 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8310161828994751 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7910484075546265 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 particle/accuracy_2_of_5‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.34736403822898865 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06151837855577469 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04404757544398308 (particle/accuracy‚0.8483498692512512 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1543874591588974 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_2_of_2‚0.9998950958251953 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8109647631645203 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6778841614723206 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 jet/accuracy_2_of_5‚1 )loss/val_total_loss‚9.836121938803917 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3425925672054291 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4015151858329773 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9624999761581421 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 trainer/global_step‚6407 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13474181294441223 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7988705039024353 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8328319787979126 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8565356731414795 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 epoch‚7 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.40513551235198975 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.16628582775592804 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03294393792748451 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 % _timestamp‚1.7845044884423103e+09 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10287167876958847 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6944962739944458 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6363696455955505 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5763193964958191 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5274940729141235 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.06449002027511597 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7389588356018066 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5902984738349915 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4739583432674408 +particle/sensitivity‚0.6258069276809692 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7401707768440247 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8"Purity/*FRt2SRqqt0FBt/event_purity‚0.185700461268425 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.505354106426239 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9835184812545776 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9859241247177124 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6363696455955505 particle/accuracy_1_of_3‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06448829919099808 +jet/accuracy_3_of_3‚0.06294076889753342 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5000000596046448 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3736170828342438 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4567784368991852 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9927241206169128 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7400000095367432 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9194021821022034 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9992063641548157 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6844088435173035 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0869923084974289 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7878133654594421 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7564361691474915 jet/accuracy_3_of_5‚1 )jet/accuracy_1_of_4‚0.959182858467102 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7266584634780884 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 /particle/accuracy_3_of_3‚0.9990074634552002 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9951246976852417 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4650000035762787 *jet/accuracy_1_of_3‚0.9123702049255371 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07500000298023224 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3758109509944916 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 _runtime‚2642.507556654 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8797886967658997 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18236763775348663 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9821010828018188 particle/accuracy_1_of_2‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7019087672233582 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.41636547446250916 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7542037963867188 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6250864863395691 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6864348649978638 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.22565880417823792‚EKò¶Î{ :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03294393792748451 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 % _timestamp‚1.7845044884423103e+09 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.10287167876958847 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.6499999761581421 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.6944962739944458 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6363696455955505 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5763193964958191 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5274940729141235 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.06449002027511597 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7389588356018066 9"Purity/1FRt15†ªâ8SRqqt1FBt/SRqqt_purity‚0.5902984738349915 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4739583432674408 +particle/sensitivity‚0.6258069276809692 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7401707768440247 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8"Purity/*FRt2SRqqt0FBt/event_purity‚0.185700461268425 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.505354106426239 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9835184812545776 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9859241247177124 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6363696455955505 particle/accuracy_1_of_3‚1 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.06448829919099808 +jet/accuracy_3_of_3‚0.06294076889753342 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5000000596046448 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3736170828342438 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4567784368991852 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9927241206169128 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.7400000095367432 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9194021821022034 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9992063641548157 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6844088435173035 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0869923084974289 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7878133654594421 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7564361691474915 jet/accuracy_3_of_5‚1 )jet/accuracy_1_of_4‚0.959182858467102 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7266584634780884 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 /particle/accuracy_3_of_3‚0.9990074634552002 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9951246976852417 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4650000035762787 *jet/accuracy_1_of_3‚0.9123702049255371 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07500000298023224 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3758109509944916 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 _runtime‚2642.507556654 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8797886967658997 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.18236763775348663 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9821010828018188 particle/accuracy_1_of_2‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7019087672233582 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.41636547446250916 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7542037963867188 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6250864863395691 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6864348649978638 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9978354573249817 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.22565880417823792 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1485610008239746 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 particle/accuracy_1_of_1‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1543874591588974 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9409578442573547 particle/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5306097865104675 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.4885680377483368 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.22363126277923584 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9939238429069519 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6274589896202087 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4014301300048828 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5421776175498962 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.13023458421230316 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.14701350033283234 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.24526335299015045 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6243506669998169 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07500000298023224 particle/accuracy_1_of_5‚1 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.532256007194519 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.01769634522497654 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13686801493167877 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15461890399456024 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6345404982566833 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_2_of_3‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8021367192268372 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6751890778541565 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.1200888380408287 jet/accuracy_4_of_5‚0 _step‚519 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.6885814666748047 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.06150821968913078 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6132687330245972 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 *jet/accuracy_1_of_1‚0.3018118441104889 7validation_average_jet_accuracy‚0.34753134846687317 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9808877110481262 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.4678795635700226 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7428649663925171 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_4‚1 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2777777910232544 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0005668934318237007 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.01895691268146038 *jet/accuracy_2_of_2‚0.1366782784461975 particle/accuracy_2_of_4‚1 particle/accuracy_5_of_5‚0 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7467973232269287 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.20371100306510925 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6555129885673523 :"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.46174076199531555 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1701967418193817 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5174086689949036 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6261477470397949 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.035790491849184036 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15499350428581238 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.049154382199048996 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.10304297506809235 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7356455326080322 particle/accuracy_3_of_4‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9642857313156128 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 'particle/f_score‚0.6392379999160767 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9940814971923828 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9642021656036377 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4388047158718109 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11562500149011612 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0714285746216774 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0714285746216774 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 *jet/accuracy_2_of_4‚0.7985166907310486 +particle/specificity‚0.9092022776603699 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_4_of_5‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15402621030807495 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2819328010082245 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0005668934318237007 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.21989865601062775 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.830365777015686 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.830365777015686 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.4359224736690521 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6215733289718628 ,jet/accuracy_4_of_4‚0.014682539738714695 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9988662004470825 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9953886866569519 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9953965544700623 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11940373480319977 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9060606956481934 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17389948666095734 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.049154382199048996 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 7"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.73295658826828 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.6666666865348816 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16613391041755676 +jet/accuracy_3_of_4‚0.08360614627599716 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8241226673126221 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.7400000095367432 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9995591044425964 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.48890626430511475 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.27272728085517883 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.010032395832240582 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.14958932995796204 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1670842468738556 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.995820164680481 )jet/accuracy_1_of_2‚0.768553614616394 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8310161828994751 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7910484075546265 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 particle/accuracy_2_of_5‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.34736403822898865 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.06151837855577469 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04404757544398308 (particle/accuracy‚0.8483498692512512 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1543874591588974 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /particle/accuracy_2_of_2‚0.9998950958251953 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8109647631645203 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6778841614723206 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 jet/accuracy_2_of_5‚1 )loss/val_total_loss‚9.836121938803917 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3425925672054291 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4015151858329773 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9624999761581421 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 trainer/global_step‚6407 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13474181294441223 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.7988705039024353 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8328319787979126 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8565356731414795 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 jet/accuracy_5_of_5‚0 epoch‚7 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.40513551235198975 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.16628582775592804‡7„ý˜g·‚(:Ü ´ÁõÒчƿ' proc.memory.availableMB‚ 191.5390625 disk.md0.in‚0 disk.dm-0.in‚ 101.51953125 disk.rbd0.out‚0# memory_percent‚98.8309383392334* proc.memory.percent‚17.533421516418457 proc.cpu.threads‚17 disk.dm-1.in‚0& disk.nvme1c1n1.out‚4132.5986328125# proc.memory.rssMB‚ 2872.67578125 cpu‚26.51740263309659& disk.nvme0c0n1.out‚14696.287109375 network.recv‚766767) disk./.usagePercent‚75.56891876951217% disk./.usageGB‚1160.1718254089355 disk.md0.out‚ 79.52734375 disk.rbd0.in‚0 disk.nvme0n1.in‚91.03125! disk.nvme1c1n1.in‚ 10.48828125 network.sent‚ 8.448266e+06! disk.dm-0.out‚16373.158203125! disk.dm-1.out‚2286.0478515625 disk.nvme0c0n1.in‚91.03125$ disk.nvme0n1.out‚14696.287109375 disk.nvme1n1.in‚ 10.48828125$ disk.nvme1n1.out‚4132.5986328125T)7›¸‚(: ´ÁõÒíüõ¿ gpu.0.gpu‚51.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚182.497' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚60.83233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬u2¹- _wandbruntime‚2659  _runtime‚2659y¿Ô‚ǺÁ _runtime‚2659.172242299 _step‚520 & lr-AdamW/pg1‚0.0001064948831213699 & lr-AdamW/pg2‚0.0001064948831213699 trainer/global_step‚6449 % _timestamp‚1.7845045051090918e+09³änÌ»Æ % _timestamp‚1.7845045051090918e+09 _runtime‚2659.172242299 _step‚520 & lr-AdamW/pg1‚0.0001064948831213699 & lr-AdamW/pg2‚0.0001064948831213699 trainer/global_step‚6449ˆ…5½ª2¼- _wandbruntime‚2659  _runtime‚2659 "‹²×'½ÑO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4527513837942 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚242327648138240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65426356 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚253345487594496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4541691961344 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚253345382533120 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44996599 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚237653660816384 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65426356 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4541603939328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚242327859746816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16571019 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚250109426966636 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49124183 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚253345382533120 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65416213 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚250109530154470 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49124181 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49124181 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4527427689468 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4674174026752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 % _timestamp‚1.7845045051099453e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16571272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚248803795633152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4541603939328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 trainer/global_step‚6449 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚248803795633152 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114540394 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚248803778593792 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚253345382533120 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44996599 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚237653660816384 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65426356 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4541603939328 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚242327859746816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16571019 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚250109426966636 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49124183 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚253345382533120 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65416213 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚250109530154470 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49124181 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49124181 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4527427689468 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4674174026752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 % _timestamp‚1.7845045051099453e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16571272 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚248803795633152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4541603939328 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 trainer/global_step‚6449 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚248803795633152 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114540394 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚248803778593792 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4527513837942 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚242327648138240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65426356 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚521 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚253345487594496 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4541691961344 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0‰ñB$à2¿- _wandbruntime‚2659  _runtime‚2659([¬Yâ&ÀÜM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚253383532078592 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49131504 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45003329 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114567772 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45003075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16573712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚245618888384288 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 % _timestamp‚1.7845045053731487e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚248841162000384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚253383427010048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 trainer/global_step‚6449 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65426111 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4528189789498 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _step‚522 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚242364252264960 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114557613 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚248841162000384 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4542370078208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65436268 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114557613 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚250146974985840 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4542282049024 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4542282049024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28429617 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚250147078173786 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4528103640912 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 114567772 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4674686787072 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16573461 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104ÒA›hç&ÁáM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16573461 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚253383532078592 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49131504 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45003329 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114567772 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45003075 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16573712 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚245618888384288 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 % _timestamp‚1.7845045053731487e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚248841162000384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚253383427010048 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 trainer/global_step‚6449 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65426111 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4528189789498 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _step‚522 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚242364252264960 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114557613 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚248841162000384 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4542370078208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65436268 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 114557613 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚250146974985840 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4542282049024 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4542282049024 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28429617 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚250147078173786 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4528103640912 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 114567772 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4674686787072Š‚¯µã2Â- _wandbruntime‚2659  _runtime‚2659<¥Bâ % _timestamp‚1.7845045053742278e+09 _runtime‚2659.437082268 1loss/SRqqt1/detection_loss‚1.0604366525670792 &loss/total_loss‚10.174796960718517 _step‚523 /loss/FBt1/detection_loss‚0.8823982292670633 trainer/global_step‚6449 0loss/FRt2/assignment_loss‚0.9475310239599738 .loss/FRt1/detection_loss‚0.556206393820569 /loss/FRt2/detection_loss‚1.4611574678234893 /loss/FBt2/detection_loss‚0.7007816649591125 epoch‚8 /loss/FRt1/assignment_loss‚2.386535933071917 2loss/SRqqt1/assignment_loss‚0.5970987469380629 2loss/SRqqt2/assignment_loss‚0.7317046994433808 1loss/FBt1/assignment_loss‚0.16355656715409697 3loss/FBt2/assignment_loss‚0.0022621642732447026 1loss/SRqqt2/detection_loss‚0.6851274174405262ÄçPΠć &loss/total_loss‚10.174796960718517 _step‚523 /loss/FBt1/detection_loss‚0.8823982292670633 trainer/global_step‚6449 0loss/FRt2/assignment_loss‚0.9475310239599738 .loss/FRt1/detection_loss‚0.556206393820569 /loss/FRt2/detection_loss‚1.4611574678234893 /loss/FBt2/detection_loss‚0.7007816649591125 epoch‚8 /loss/FRt1/assignment_loss‚2.386535933071917 2loss/SRqqt1/assignment_loss‚0.5970987469380629 2loss/SRqqt2/assignment_loss‚0.7317046994433808 1loss/FBt1/assignment_loss‚0.16355656715409697 3loss/FBt2/assignment_loss‚0.0022621642732447026 1loss/SRqqt2/detection_loss‚0.6851274174405262 % _timestamp‚1.7845045053742278e+09 _runtime‚2659.437082268 1loss/SRqqt1/detection_loss‚1.0604366525670792‹eKÁÒkÅ‚(:à ÃÁõÒ®õÝ¿ disk.nvme0n1.in‚91.03125! disk.nvme1c1n1.in‚ 10.48828125 disk.nvme1n1.in‚ 10.48828125 disk.rbd0.in‚0' proc.memory.availableMB‚ 191.5078125& disk.nvme0n1.out‚14697.70263671875 disk.rbd0.out‚0# proc.memory.rssMB‚ 2872.67578125 network.recv‚770490( disk./.usagePercent‚75.5534328752029& disk.nvme1c1n1.out‚4213.0810546875 disk.md0.in‚0" disk.dm-1.out‚2365.45849609375 disk.nvme0c0n1.in‚91.03125( disk.nvme0c0n1.out‚14697.70263671875$ memory_percent‚98.83112907409668% disk./.usageGB‚1159.9340782165527$ disk.nvme1n1.out‚4213.0810546875 disk.md0.out‚ 80.4921875 disk.dm-0.in‚ 101.51953125 disk.dm-1.in‚0* proc.memory.percent‚17.533421516418457 cpu‚26.51413467107442 proc.cpu.threads‚17! disk.dm-0.out‚16373.619140625 network.sent‚ 8.506353e+06¸S9s›Æ‚(: ÃÁõÒìׇÀ gpu.0.gpu‚82.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚200.51' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.83666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01^Œø2Ç- _wandbruntime‚2678  _runtime‚26788[àÇÈÁ _runtime‚ 2678.3698588 _step‚524 ' lr-AdamW/pg1‚0.00010639646460066954 ' lr-AdamW/pg2‚0.00010639646460066954 trainer/global_step‚6499 % _timestamp‚1.7845045243067455e+09ÔãkÌÉÆ trainer/global_step‚6499 % _timestamp‚1.7845045243067455e+09 _runtime‚ 2678.3698588 _step‚524 ' lr-AdamW/pg1‚0.00010639646460066954 ' lr-AdamW/pg2‚0.00010639646460066954Œ³ªÀ†2Ê- _wandbruntime‚2678  _runtime‚2678{€Ó9’ËÒO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49490233 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49490231 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45334673 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 _step‚525 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 % _timestamp‚1.7845045243076537e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4575629703168 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4561343313350 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚255247743697920 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 _runtime‚2678.370739381 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 115403196 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4575541681152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16694846 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65923108 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65912965 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255247638636544 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65923108 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251986859830044 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚239438418678784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244147613666304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚251986963017878 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4561257164876 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28639824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49490231 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚244147410446336 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚255247743697920 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28639827 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚239438443582464 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚250672096955392 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚250672113994752 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16694602 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚250672113994752 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 115413341 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚65912965 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚250672096955392 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4561343313350 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚255247743697920 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 _runtime‚2678.370739381 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 115403196 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4575541681152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16694846 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65923108 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚65912965 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚255247638636544 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚65923108 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251986859830044 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚239438418678784 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244147613666304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚251986963017878 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4561257164876 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28639824 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49490231 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚244147410446336Hkªœ2Í- _wandbruntime‚2678  _runtime‚2678Õi¾Žâ&ÎÜM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚250709480361984 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45341419 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28644114 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4709872376320 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45341160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4709691970048 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115430574 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49497552 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚244184011116032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚252024511037194 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115420415 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚526 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚247462491772288 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49497552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16697302 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115420415 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65933020 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚239474113836032 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65922863 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚250709480361984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4576219790848 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚252024407849248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 trainer/global_step‚6499 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49497554 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚255285683113472 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4576219790848 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚2678.629221871 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4576307820032 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚244183805806080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115430574 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4576307820032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255285788182016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16697046 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4561933116320 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚250709463322624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28644117 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4562019264906 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65922863 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚250709463322624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7845045245659285e+09 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚255285788182016 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚239474138739712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65933020 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚247462474732928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49497554 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152nQoÚ¦ÏáM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚49497552 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚244184011116032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚252024511037194 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 115420415 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚526 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚247462491772288 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49497552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16697302 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 115420415 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65933020 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚239474113836032 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚65922863 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚250709480361984 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4576219790848 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚252024407849248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 trainer/global_step‚6499 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49497554 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚255285683113472 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4576219790848 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚2678.629221871 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4576307820032 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚244183805806080 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 115430574 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4576307820032 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚255285788182016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16697046 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4561933116320 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚250709463322624 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28644117 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4562019264906 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65922863 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚250709463322624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7845045245659285e+09 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚Þ/úA 255285788182016 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚239474138739712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65933020 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚247462474732928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49497554 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚250709480361984 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45341419 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28644114 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4709872376320 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45341160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4709691970048 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 115430574Ž·CYz2Ð- _wandbruntime‚2678  _runtime‚2678sí»Ñþ 0loss/FRt2/assignment_loss‚1.0716988833870476 2loss/SRqqt1/assignment_loss‚0.6621891090248759 _runtime‚2678.629800972 /loss/FRt1/assignment_loss‚2.654413053393364 /loss/FBt1/detection_loss‚0.8656647339230403 epoch‚8 2loss/SRqqt2/assignment_loss‚0.6341217037126694 /loss/FRt1/detection_loss‚0.5528192586871623 1loss/SRqqt1/detection_loss‚1.0212876552261576 1loss/SRqqt2/detection_loss‚0.6523417562002823 %loss/total_loss‚10.21706022745632 trainer/global_step‚6499 % _timestamp‚1.7845045245668647e+09 0loss/FBt1/assignment_loss‚0.0998966225210296 1loss/FBt2/assignment_loss‚0.07924931252790748 /loss/FRt2/detection_loss‚1.4025981124769162 .loss/FBt2/detection_loss‚0.520780026375869 _step‚527œwð Òƒ epoch‚8 2loss/SRqqt2/assignment_loss‚0.6341217037126694 /loss/FRt1/detection_loss‚0.5528192586871623 1loss/SRqqt1/detection_loss‚1.0212876552261576 1loss/SRqqt2/detection_loss‚0.6523417562002823 %loss/total_loss‚10.21706022745632 trainer/global_step‚6499 % _timestamp‚1.7845045245668647e+09 0loss/FBt1/assignment_loss‚0.0998966225210296 1loss/FBt2/assignment_loss‚0.07924931252790748 /loss/FRt2/detection_loss‚1.4025981124769162 .loss/FBt2/detection_loss‚0.520780026375869 _step‚527 0loss/FRt2/assignment_loss‚1.0716988833870476 2loss/SRqqt1/assignment_loss‚0.6621891090248759 _runtime‚2678.629800972 /loss/FRt1/assignment_loss‚2.654413053393364 /loss/FBt1/detection_loss‚0.8656647339230403ŇämÓ‚(:â ÒÁõÒà—¢À( disk.nvme0c0n1.out‚14701.90185546875' disk.nvme1c1n1.out‚4217.51220703125 disk.nvme1n1.in‚ 10.48828125 disk.dm-1.in‚0$ proc.memory.availableMB‚191.4375 proc.cpu.threads‚17 network.sent‚ 8.581682e+06) disk./.usagePercent‚75.55372557758876% disk./.usageGB‚1159.9385719299316 disk.nvme0n1.in‚91.03125 disk.md0.out‚ 81.2421875# disk.dm-0.out‚16378.34423828125" disk.dm-1.out‚2367.79052734375* proc.memory.percent‚17.533421516418457 disk.rbd0.in‚0& disk.nvme0n1.out‚14701.90185546875! disk.nvme1c1n1.in‚ 10.48828125 disk.dm-0.in‚ 101.51953125$ memory_percent‚98.83155822753906 cpu‚26.51071060163123 disk.nvme0c0n1.in‚91.03125% disk.nvme1n1.out‚4217.51220703125 disk.md0.in‚0# proc.memory.rssMB‚ 2872.67578125 network.recv‚774612 disk.rbd0.out‚0?l—y›Ô‚(: ÒÁõÒЫÌÀ gpu.0.gpu‚25.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚198.962' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.32066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥–ž2Õ- _wandbruntime‚2697  _runtime‚2697Êø“GÈÖ ' lr-AdamW/pg1‚0.00010629723989550854 ' lr-AdamW/pg2‚0.00010629723989550854 trainer/global_step‚6549 $ _timestamp‚1.784504542972893e+09 _runtime‚2697.036029453 _step‚528aE2Í×Ç ' lr-AdamW/pg2‚0.00010629723989550854 trainer/global_step‚6549 $ _timestamp‚1.784504542972893e+09 _runtime‚2697.036029453 _step‚528 ' lr-AdamW/pg1‚0.00010629723989550854'bÒz2Ø- _wandbruntime‚2697  _runtime‚2697ÔMlc×'ÙÑO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 _runtime‚2697.040620531 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16817945 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚245967366712320 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚245967155103744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚257149999801344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4609567444992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16818210 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116276143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚241223176541184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66409717 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4744165267456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4743978562560 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 _step‚529 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚249269206053168 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49856283 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4609479422976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚252540415316992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28854327 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚249269223092528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116265998 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚252540432356352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45672537 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49856281 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4595086640284 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4609567444992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66419860 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66409717 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4609479422976 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚257149894739968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚257149894739968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28854324 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016;˜ÿ9Ü'ÚÖO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66409717 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4609479422976 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚257149894739968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚257149894739968 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28854324 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 _runtime‚2697.040620531 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16817945 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚245967366712320 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚245967155103744 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚257149999801344 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4609567444992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16818210 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116276143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚241223176541184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66409717 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4744165267456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4743978562560 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 _step‚529 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚249269206053168 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49856283 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4609479422976 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚252540415316992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚28854327 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚249269223092528 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116265998 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚252540432356352 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45672537 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚49856281 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4595086640284 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4609567444992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66419860 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482‘SK02Û- _wandbruntime‚2697  _runtime‚2697úsL8ÜÚM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116283217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28858617 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45679005 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚253901943900602 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66419615 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16820391 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66429772 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116293376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49863604 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16820662 $ _timestamp‚1.784504543234304e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4595848740314 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚252577781684224 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚246003548153344 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚249306095160288 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 2697.29760382 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_baté㨠tch_end/allocated_bytes.all.current‚ 105068544 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116283217 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45679279 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚257188044285440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4595762591728 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4610245561856 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚257187939216896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚252577781684224 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚252577798723584 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116293376 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49863602 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66429772 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66419615 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚249306078120928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 trainer/global_step‚6549 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4610157532672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28858614 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0ÓÚ—Ãå&ÝßM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚252577781684224 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚252577798723584 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 116293376 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚49863602 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66429772 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66419615 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚249306078120928 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 trainer/global_step‚6549 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4610157532672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28858614 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 116283217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28858617 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45679005 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚253901943900602 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66419615 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16820391 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66429772 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116293376 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚49863604 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16820662 $ _timestamp‚1.784504543234304e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4595848740314 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚252577781684224 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚246003548153344 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚249306095160288 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 2697.29760382 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116283217 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45679279 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚257188044285440 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4595762591728 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4610245561856 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚257187939216896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0’ÏŒàï2Þ- _wandbruntime‚2697  _runtime‚2697:@5Ê߀ _step‚531 1loss/SRqqt2/detection_loss‚0.6372753868384965 /loss/FBt2/detection_loss‚0.4968054372307606 /loss/FRt1/assignment_loss‚2.446768322198287 2loss/SRqqt1/assignment_loss‚0.7997711806891173 2loss/FBt2/assignment_loss‚0.012998231785432917 /loss/FBt1/detection_loss‚1.1018069567420752 &loss/total_loss‚10.359260689212277 epoch‚8 % _timestamp‚1.7845045432352817e+09 /loss/FRt1/detection_loss‚0.5690243775102436 /loss/FRt2/detection_loss‚1.3301081323499262 0loss/SRqqt1/detection_loss‚1.277046083499861 trainer/global_step‚6549 /loss/FRt2/assignment_loss‚1.005532919939331 2loss/SRqqt2/assignment_loss‚0.6309556581036889 1loss/FBt1/assignment_loss‚0.05116800232505767 _runtime‚2697.298089561K×2^ à… 2loss/SRqqt1/assignment_loss‚0.7997711806891173 2loss/FBt2/assignment_loss‚0.012998231785432917 /loss/FBt1/detection_loss‚1.1018069567420752 &loss/total_loss‚10.359260689212277 epoch‚8 % _timestamp‚1.7845045432352817e+09 /loss/FRt1/detection_loss‚0.5690243775102436 /loss/FRt2/detection_loss‚1.3301081323499262 0loss/SRqqt1/detection_loss‚1.277046083499861 trainer/global_step‚6549 /loss/FRt2/assignment_loss‚1.005532919939331 2loss/SRqqt2/assignment_loss‚0.6309556581036889 1loss/FBt1/assignment_loss‚0.05116800232505767 _runtime‚2697.298089561 _step‚531 1loss/SRqqt2/detection_loss‚0.6372753868384965 /loss/FBt2/detection_loss‚0.4968054372307606 /loss/FRt1/assignment_loss‚2.446768322198287“õnN|›á‚(: áÁõÒ¤Ô«À gpu.0.gpu‚57.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚195.557' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.18566666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06½²Ahâ‚(:Ý áÁõÒ¡÷»Á network.sent‚ 8.630054e+06! disk.nvme1c1n1.in‚ 10.48828125 disk.md0.out‚ 82.4609375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚91.03125 cpu‚26.507679298644547 network.recv‚778781) disk./.usagePercent‚75.68477002597169" disk.nvme1n1.out‚ 4224.88671875! disk.dm-1.out‚2371.0244140625 disk.nvme0n1.in‚91.03125% disk./.usageGB‚1161.9504318237305 disk.md0.in‚0 disk.dm-0.in‚ 101.51953125# disk.dm-0.out‚16384.93408203125( disk.nvme0c0n1.out‚14706.88232421875& disk.nvme0n1.out‚14706.88232421875$ memory_percent‚98.83174896240234% proc.memory.availableMB‚ 191.40625$ disk.nvme1c1n1.out‚ 4224.88671875 disk.nvme1n1.in‚ 10.48828125 disk.dm-1.in‚0# proc.memory.rssMB‚ 2872.67578125* proc.memory.percent‚17.533421516418457 proc.cpu.threads‚17ÆóEcã‚(:Ø ðÁõÒ²ÊÍ¿% disk.nvme1c1n1.out‚4228.208984375 disk.nvme1n1.in‚ 10.48828125# disk.nvme1n1.out‚4228.208984375 cpu‚26.50446869274248 disk.md0.in‚0 disk.dm-0.in‚ 101.51953125# memory_percent‚98.8332748413086% proc.memory.availableMB‚ 191.15625 disk.md0.out‚ 83.6640625 network.sent‚ 8.69433e+06 network.recv‚783571 disk.nvme0n1.in‚91.03125! disk.dm-1.out‚2371.1259765625 disk.rbd0.out‚0 disk.nvme0c0n1.in‚91.03125 proc.cpu.threads‚17% disk./.usageGB‚1162.7120361328125! disk.nvme1c1n1.in‚ 10.48828125! disk.dm-0.out‚18316.947265625 disk.dm-1.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚16638.1552734375# proc.memory.rssMB‚ 2872.67578125* proc.memory.percent‚17.533421516418457) disk./.usagePercent‚75.73437786242062% disk.nvme0n1.out‚16638.1552734375X|Öœä‚(:‘ ðÁõÒõ—ø¿ gpu.0.gpu‚63.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚156.121' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚52.040333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:rŽŸ2å- _wandbruntime‚2716  _runtime‚2716°I=Èæ trainer/global_step‚6599 $ _timestamp‚1.784504561987971e+09 _runtime‚2716.052578445 _step‚532 ' lr-AdamW/pg1‚0.00010619721059517193 ' lr-AdamW/pg2‚0.00010619721059517193óµëBÍçÇ $ _timestamp‚1.784504561987971e+09 _runtime‚2716.052578445 _step‚532 ' lr-AdamW/pg1‚0.00010619721059517193 ' lr-AdamW/pg2‚0.00010619721059517193 trainer/global_step‚6599”¸†Âá2è- _wandbruntime‚2716  _runtime‚2716ñÄ5‚Ø'éÒO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚243007959307264 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50222333 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50222333 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚247786900683776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259052150843392 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4778966280192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117128800 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50222331 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66906469 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29068827 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚255741828744694 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66916612 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚243007934403584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259052255904768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚251112826480528 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚255741725556860 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚254408733678592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4643505186816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46009434 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0Ù ÿôê×O A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚254408733678592 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4643505186816 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46009434 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚243007959307264 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50222333 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50222333 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚247786900683776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259052150843392 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4778966280192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117128800 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50222331 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66906469 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29068827 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚255741828744694 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66916612 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚243007934403584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259052255904768 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚251112826480528 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚255741725556860•Ìñ[«2ë- _wandbruntime‚2716  _runtime‚2716/õ"Øâ&ìÜM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4629592067136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66926524 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66916367 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚255779273576064 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚259090300388864 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚247823501549056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚243043629560832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚254446100045824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4644183303680 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4644095274496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46016173 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4644183303680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚259090300388864 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117156178 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16943314 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚251149698548288 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251149681508928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46016431 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29073117 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117146019 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚259090195320320 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚254446117085184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚247823296239104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50229652 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584ä´)ç&íáM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46016173 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4644183303680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚259090300388864 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117156178 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16943314 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚251149698548288 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251149681508928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46016431 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29073117 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117146019 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚259090195320320 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚254446117085184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚247823296239104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50229652 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4629592067136 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66926524 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66916367 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚255779273576064 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚259090300388864 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚247823501549056 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚243043629560832 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚254446100045824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4644183303680 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4644095274496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0–Phðt2î- _wandbruntime‚2716  _runtime‚2716l–”ï€ 0loss/FRt2/assignment_loss‚0.9133880598872316 /loss/FRt2/detection_loss‚1.4996293090370612 /loss/FBt2/detection_loss‚0.6479039640417175 &loss/total_loss‚10.804640462414687 trainer/global_step‚6599 2loss/SRqqt1/assignment_loss‚0.6788698458142248 2loss/FBt2/assignment_loss‚0.003658448555724034 /loss/FRt1/detection_loss‚0.6006974882690386 0loss/SRqqt1/detection_loss‚1.157631620853149 _step‚535 2loss/SRqqt2/assignment_loss‚0.6678278128946137 epoch‚8 $ _timestamp‚1.784504562250013e+09 /loss/FRt1/assignment_loss‚2.620642346976337 1loss/FBt1/assignment_loss‚0.16592724608188275 1loss/SRqqt2/detection_loss‚0.7104318060469421 /loss/FBt1/detection_loss‚1.1380325139567644 _runtime‚2716.312833835#Fe ð… 0loss/FRt2/assignment_loss‚0.9133880598872316 /loss/FRt2/detection_loss‚1.4996293090370612 /loss/FBt2/detection_loss‚0.6479039640417175 &loss/total_loss‚10.804640462414687 trainer/global_step‚6599 2loss/SRqqt1/assignment_loss‚0.6788698458142248 2loss/FBt2/assignment_loss‚0.003658448555724034 /loss/FRt1/detection_loss‚0.6006974882690386 0loss/SRqqt1/detection_loss‚1.157631620853149 _step‚535 2loss/SRqqt2/assignment_loss‚0.6678278128946137 epoch‚8 $ _timestamp‚1.784504562250013e+09 /loss/FRt1/assignment_loss‚2.620642346976337 1loss/FBt1/assignment_loss‚0.16592724608188275 1loss/SRqqt2/detection_loss‚0.7104318060469421 /loss/FBt1/detection_loss‚1.1380325139567644 _runtime‚2716.312833835—2…Õnñ‚(:ã ÿÁõÒ‹¡‘À disk.nvme0c0n1.in‚91.03125 cpu‚26.501335986346987! disk.dm-0.out‚18324.130859375& disk.nvme0n1.out‚16646.20361328125& disk.nvme1c1n1.out‚4229.7001953125 disk.nvme1n1.in‚ 10.48828125 disk.md0.out‚ 84.5546875( disk.nvme0c0n1.out‚16646.20361328125$ memory_percent‚98.83196353912354# proc.memory.rssMB‚ 2872.67578125 disk.md0.in‚0" disk.dm-1.out‚2371.61962890625( proc.memory.availableMB‚ 191.37109375* proc.memory.percent‚17.533421516418457 proc.cpu.threads‚17 network.sent‚ 8.710095e+06 network.recv‚786624 disk.nvme0n1.in‚91.03125! disk.nvme1c1n1.in‚ 10.48828125 disk.rbd0.in‚0 disk.rbd0.out‚0) disk./.usagePercent‚75.73545077320341% disk./.usageGB‚1162.7285079956055$ disk.nvme1n1.out‚4229.7001953125 disk.dm-0.in‚ 101.51953125 disk.dm-1.in‚0ÒÆZšò‚(: ÿÁõÒÿª¿À gpu.0.gpu‚12.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚195.25' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.08333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ?Qr2ó- _wandbruntime‚2735  _runtime‚2735!»dÅô¿ % lr-AdamW/pg1‚0.000106096378301832 % lr-AdamW/pg2‚0.000106096378301832 trainer/global_step‚6649 % _timestamp‚1.7845045810213463e+09 _runtime‚2735.085591762 _step‚53687T*ÊõÄ _runtime‚2735.085591762 _step‚536 % lr-AdamW/pg1‚0.000106096378301832 % lr-AdamW/pg2‚0.000106096378301832 trainer/global_step‚6649 % _timestamp‚1.7845045810213463e+09˜¼¦ú­2ö- _wandbruntime‚2735  _runtime‚2735=aä¾| ÷ÒO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67403221 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4677442928640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚257619158420268 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67413364 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29283327 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚244792692265984 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4677442928640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚252956429868528 _runtime‚2735.085648042 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚252956412829168 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4814113705984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚260954406946816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46347160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50588383 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50588383 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚249606623461376 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚260954512008192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚260954512008192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚249606830875648 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4677354906624 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚256277069079552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚260954406946816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152þ¤sÌÝ'ø×O cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4814113705984 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚260954406946816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46347160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50588383 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50588383 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚249606623461376 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚260954512008192 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚260954512008192 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚249606830875648 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4677354906624 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚256277069079552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚260954406946816 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67403221 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4677442928640 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚257619158420268 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67413364 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29283327 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚244792692265984 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4677442928640 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚252956429868528 _runtime‚2735.085648042 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚252956412829168 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2™Y w2ù- _wandbruntime‚2735  _runtime‚2735ÀØž¤á&úÛM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46353893 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4663507691130 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50595704 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚6649 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67413119 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67423276 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46354124 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚228 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4814634920448 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚260992451423744 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50595702 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚257656706439472 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _step‚538 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118018980 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚252993301936288 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚256314435446784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 % _timestamp‚1.7845045812813468e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4678033016320 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17066507 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 2735.34473636 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67423276 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚252993284896928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4663421542544 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 118018980 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚256314418407424 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118008821 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50595704 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚260992556492288 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67413119 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚244828412326912 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244828387423232 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚260992556492288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29287617 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17066279 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4678121045504 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚256314418407424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29287614 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4814811132416 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚260992556492288 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67413119 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚244828412326912 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244828387423232 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚260992556492288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29287617 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17066279 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4678121045504 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚256314418407424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29287614 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4814811132416 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46353893 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4663507691130 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50595704 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_ïP¡Ubatch_end/segment.all.freed‚0 trainer/global_step‚6649 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67413119 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67423276 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚46354124 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚228 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4814634920448 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚260992451423744 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50595702 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚257656706439472 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _step‚538 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118018980 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚252993301936288 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚256314435446784 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 % _timestamp‚1.7845045812813468e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4678033016320šÅ“ÜÉ2ü- _wandbruntime‚2735  _runtime‚2735)Ü€ý 1loss/FBt2/assignment_loss‚0.00169213482031519 /loss/FRt1/detection_loss‚0.5695580042439428 1loss/SRqqt1/detection_loss‚1.2866248461975962 /loss/FBt2/detection_loss‚0.5790871192804228 trainer/global_step‚6649 _runtime‚2735.345267871 0loss/FRt2/assignment_loss‚1.1003147409689473 2loss/SRqqt1/assignment_loss‚0.7582179900714169 /loss/FRt2/detection_loss‚1.4593743419932725 epoch‚8 1loss/SRqqt2/detection_loss‚0.7289621006711755 /loss/FBt1/detection_loss‚0.7883554584889984 $ _timestamp‚1.784504581282423e+09 0loss/FRt1/assignment_loss‚2.4619424803690477 2loss/SRqqt2/assignment_loss‚0.6263208464265984 1loss/FBt1/assignment_loss‚0.07283559596367392 &loss/total_loss‚10.433285659495407 _step‚539‚—ñ4 þ† 0loss/FRt2/assignment_loss‚1.1003147409689473 2loss/SRqqt1/assignment_loss‚0.7582179900714169 /loss/FRt2/detection_loss‚1.4593743419932725 epoch‚8 1loss/SRqqt2/detection_loss‚0.7289621006711755 /loss/FBt1/detection_loss‚0.7883554584889984 $ _timestamp‚1.784504581282423e+09 0loss/FRt1/assignment_loss‚2.4619424803690477 2loss/SRqqt2/assignment_loss‚0.6263208464265984 1loss/FBt1/assignment_loss‚0.07283559596367392 &loss/total_loss‚10.433285659495407 _step‚539 1loss/FBt2/assignment_loss‚0.00169213482031519 /loss/FRt1/detection_loss‚0.5695580042439428 1loss/SRqqt1/detection_loss‚1.2866248461975962 /loss/FBt2/detection_loss‚0.5790871192804228 trainer/global_step‚6649 _runtime‚2735.345267871›}²,¤jÿ‚(:ß ŽÂõÒ؛ɿ& disk.nvme0c0n1.out‚17418.845703125* proc.memory.percent‚17.533445358276367 network.sent‚ 8.759624e+06 network.recv‚791159) disk./.usagePercent‚75.73697367042152$ disk.nvme0n1.out‚17418.845703125% disk.nvme1n1.out‚4348.43603515625! disk.dm-0.out‚19213.904296875" disk.dm-1.out‚2371.72119140625 disk.rbd0.in‚0% disk./.usageGB‚1162.7518882751465 disk.nvme0n1.in‚91.03125! disk.nvme1c1n1.in‚ 10.48828125 disk.md0.out‚ 85.265625 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚91.03125$ memory_percent‚98.83239269256592 disk.md0.in‚0( proc.memory.availableMB‚ 191.30078125" proc.memory.rssMB‚ 2872.6796875 cpu‚26.498349884490167 proc.cpu.threads‚17' disk.nvme1c1n1.out‚4348.43603515625 disk.nvme1n1.in‚ 10.48828125 disk.dm-0.in‚ 101.519531256¨„›€‚(: ŽÂõÒ±¹œÀ gpu.0.gpu‚59.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚155.874' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚51.95799999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02Q_È2- _wandbruntime‚2754  _runtime‚2754Ü9l)É‚Ã ' lr-AdamW/pg1‚0.00010599474463052258 ' lr-AdamW/pg2‚0.00010599474463052258 trainer/global_step‚6699 % _timestamp‚1.7845046002539306e+09 _runtime‚2754.317083991 _step‚540í—œ,ÎƒÈ ' lr-AdamW/pg2‚0.00010599474463052258 trainer/global_step‚6699 % _timestamp‚1.7845046002539306e+09 _runtime‚2754.317083991 _step‚540 ' lr-AdamW/pg1‚0.00010599474463052258œ®Èô2„- _wandbruntime‚2754  _runtime‚2754ªþ&gØ'…ÒO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4711380670464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67899973 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚262856768111616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17187309 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚254800016217168 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚258145370401792 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚6699 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4711292648448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚262856663050240 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46684873 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚251426310565888 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67899973 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4696575066508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚262856768111616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4711292648448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29497827 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 _runtime‚2754.321612849 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67910116 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚251426517980160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 % _timestamp‚1.7845046002549284e+09 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚262856663050240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 118864549 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚254800033256528 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50954433 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67910116 _step‚541 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4711380670464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚67899973 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚262856768111616 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17187309 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚254800016217168 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚258145370401792 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚6699 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4711292648448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚262856663050240 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚46684873 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚251426310565888 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚67899973 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4696575066508 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚262856768111616 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4711292648448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29497827 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 _runtime‚2754.321612849 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚67910116 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚251426517980160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 % _timestamp‚1.7845046002549284e+09 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚262856663050240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 118864549 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚254800033256528 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50954433 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚67910116 _step‚541 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4711970758144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4697337166538 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50961754 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4711970758144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50961752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 trainer/global_step‚6699 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚262894707527168 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46691630 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 _step‚542 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4849745158656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 118871623 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118871623 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚258182753808384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67909871 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.pea 4 ÎÄk‚ 313566208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50961752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚246613145285632 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚259534242490826 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29502117 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7845046005215058e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 118881782 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67920028 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚258182736769024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4712058787328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚246613170189312 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4712058787328 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 _runtime‚ 2754.58482491 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29502114 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚258182753808384 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118881782 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4697251017952l¼3/æ&‰àM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50961752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 trainer/global_step‚6699 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚262894707527168 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46691630 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 _step‚542 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4849745158656 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 118871623 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 118871623 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚258182753808384 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67909871 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50961752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚246613145285632 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚259534242490826 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29502117 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7845046005215058e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 118881782 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67920028 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚258182736769024 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4712058787328 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚246613170189312 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4712058787328 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 _runtime‚ 2754.58482491 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29502114 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚258182753808384 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 118881782 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4697251017952 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚251462915347968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4849554266624 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67920028 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4711970758144 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4697337166538 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50961754 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4711970758144 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0žXK!#2Š- _wandbruntime‚2754  _runtime‚2754ßÁ¼£‹€ /loss/FRt1/assignment_loss‚2.516692401536929 1loss/SRqqt1/assignment_loss‚0.800185673750327 1loss/SRqqt1/detection_loss‚1.1617352474992821 trainer/global_step‚6699 0loss/FRt2/assignment_loss‚1.0302146159588021 /loss/FRt2/detection_loss‚1.3278630129755178 1loss/SRqqt2/detection_loss‚0.7092893544350409 $ _timestamp‚1.784504600522573e+09 /loss/FRt1/detection_loss‚0.5885615191057146 /loss/FBt1/detection_loss‚0.8267462890528675 epoch‚8 2loss/SRqqt2/assignment_loss‚0.6348332268334772 1loss/FBt1/assignment_loss‚0.10588313390175992 2loss/FBt2/assignment_loss‚0.002908852772661718 /loss/FBt2/detection_loss‚1.1959658078522535 &loss/total_loss‚10.900879135674632 _runtime‚2754.588579596 _step‚543Hòq Œ… trainer/global_step‚6699 0loss/FRt2/assignment_loss‚1.0302146159588021 /loss/FRt2/detection_loss‚1.3278630129755178 1loss/SRqqt2/detection_loss‚0.7092893544350409 $ _timestamp‚1.784504600522573e+09 /loss/FRt1/detection_loss‚0.5885615191057146 /loss/FBt1/detection_loss‚0.8267462890528675 epoch‚8 2loss/SRqqt2/assignment_loss‚0.6348332268334772 1loss/FBt1/assignment_loss‚0.10588313390175992 2loss/FBt2/assignment_loss‚0.002908852772661718 /loss/FBt2/detection_loss‚1.1959658078522535 &loss/total_loss‚10.900879135674632 _runtime‚2754.588579596 _step‚543 /loss/FRt1/assignment_loss‚2.516692401536929 1loss/SRqqt1/assignment_loss‚0.800185673750327 1loss/SRqqt1/detection_loss‚1.1617352474992821ŸC±r\‚(:Ñ ÂõÒðÁ÷¿ disk.md0.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 2372.046875$ memory_percent‚98.83260726928711 cpu‚26.495107227790243# disk.nvme1n1.out‚4415.623046875 network.sent‚ 8.835908e+06 disk.rbd0.out‚0% disk.nvme1c1n1.out‚4415.623046875! disk.nvme1c1n1.in‚ 10.48828125 disk.nvme1n1.in‚ 10.48828125 disk.md0.out‚ 86.23046875 disk.dm-0.in‚ 101.51953125! disk.dm-0.out‚19462.931640625& proc.memory.availableMB‚ 191.265625" proc.memory.rssMB‚ 2872.6796875 network.recv‚795447) disk./.usagePercent‚75.74794951294322$ disk./.usageGB‚1162.920394897461 disk.nvme0c0n1.in‚91.03125" disk.nvme0n1.out‚ 17603.0390625* proc.memory.percent‚17.533445358276367 proc.cpu.threads‚17 disk.rbd0.in‚0$ disk.nvme0c0n1.out‚ 17603.0390625 disk.nvme0n1.in‚91.03125‰ê‡éŽ‚(:… ÂõÒ¬ÂæÁ gpu.0.gpu‚66.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚117.651' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚39.217 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ‚XU2- _wandbruntime‚2773  _runtime‚2773‰’óÉà _runtime‚2773.534581261 _step‚544 ' lr-AdamW/pg1‚0.00010589231120911322 ' lr-AdamW/pg2‚0.00010589231120911322 trainer/global_step‚6749 % _timestamp‚1.7845046194705079e+09SEÖÎ‘È ' lr-AdamW/pg2‚0.00010589231120911322 trainer/global_step‚6749 % _timestamp‚1.7845046194705079e+09 _runtime‚2773.534581261 _step‚544 ' lr-AdamW/pg1‚0.00010589231120911322 ôª«³2’- _wandbruntime‚2773  _runtime‚2773PiØ'“ÒO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚261374024147084 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚264758919153664 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚260013705802752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17310128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚248362232894464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4884004578304 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚256643619605168 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4730490690390 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68406868 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚248362207990784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4745318412288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚253246027961344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚260013688763392 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68396725 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51320483 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚264759024215040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 trainer/global_step‚6749 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51320483 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17310386 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4745318412288 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚2773.534661291 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4745230390272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51320481 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4730404541916 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚264759024215040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29712327 % _timestamp‚1.7845046194711652e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4745230390272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47022452 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚261374127334918 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68396725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29712324 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚545 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51320481 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚253246237472768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚260013705802752 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68406868 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4883819970560 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚264758919153664 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚260013688763392 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 119727351 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016¬ôú›ù”×O LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚264758919153664 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚260013688763392 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 119727351 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚261374024147084 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚264758919153664 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚260013705802752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17310128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚248362232894464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4884004578304 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚256643619605168 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4730490690390 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68406868 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚248362207990784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4745318412288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚253246027961344 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/o‡[JKäversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚260013688763392 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68396725 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51320483 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚264759024215040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 trainer/global_step‚6749 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51320483 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17310386 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4745318412288 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚2773.534661291 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4745230390272 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51320481 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4730404541916 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚264759024215040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29712327 % _timestamp‚1.7845046194711652e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4745230390272 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47022452 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚261374127334918 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68396725 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29712324 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚545 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51320481 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚253246237472768 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚260013705802752 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68406868 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4883819970560 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0¡?XáÖ2•- _wandbruntime‚2774  _runtime‚2774ŸÄÅâ&–ÜM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4745996529152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚261411572166288 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256680508712288 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68416780 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4731080493360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚253282422024704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚546 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29716614 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4731166641946 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4745908499968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚264797068699136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚260051055130624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29716617 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119734425 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51327804 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119744584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68416780 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47029236 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17312885 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4745996529152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68406623 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚260051072169984 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51327802 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚264797068699136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119734425 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚248397928051712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119744584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51327802 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4884697185792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 _runtime‚2774.110712862 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚260051072169984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47029502 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚261411675354234 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚256680491672928V˜íç&—áM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119734425 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚51327804 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119744584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68416780 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47029236 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17312885 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4745996529152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68406623 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚260051072169984 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚51327802 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚264797068699136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119734425 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚248397928051712 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119744584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚51327802 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4884697185792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 _runtime‚2774.110712862 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚260051072169984 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47029502 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚261411675354234 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚256680491672928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4745996529152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚261411572166288 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256680508712288 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68416780 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4731080493360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚253282422024704 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚546 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29716614 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4731166641946 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4745908499968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚264797068699136 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚260051055130624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29716617 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0¢½¬­¨2˜- _wandbruntime‚2774  _runtime‚2774Óòf%™ /loss/FBt1/detection_loss‚0.7107683960805898 /loss/FBt2/detection_loss‚0.5329890017445874 0loss/SRqqt1/detection_loss‚1.251556919142604 trainer/global_step‚6749 _runtime‚2774.111363194 _step‚547 2loss/SRqqt2/assignment_loss‚0.6922360982836994 1loss/SRqqt2/detection_loss‚0.6873232038004462 $ _timestamp‚1.784504620048476e+09 /loss/FRt1/assignment_loss‚2.454380111764618 1loss/FBt1/assignment_loss‚0.02813736886736004 &loss/total_loss‚10.161255586724565 epoch‚8 0loss/FRt2/assignment_loss‚0.9722803871786229 2loss/SRqqt1/assignment_loss‚0.7646995520666958 3loss/FBt2/assignment_loss‚0.0011039251058304381 /loss/FRt1/detection_loss‚0.6694757773976828 /loss/FRt2/detection_loss‚1.3963048452918287 FdÛ š† 0loss/FRt2/assignment_loss‚0.9722803871786229 2loss/SRqqt1/assignment_loss‚0.7646995520666958 3loss/FBt2/assignment_loss‚0.0011039251058304381 /loss/FRt1/detection_loss‚0.6694757773976828 /loss/FRt2/detection_loss‚1.3963048452918287 /loss/FBt1/detection_loss‚0.7107683960805898 /loss/FBt2/detection_loss‚0.5329890017445874 0loss/SRqqt1/detection_loss‚1.251556919142604 trainer/global_step‚6749 _runtime‚2774.111363194 _step‚547 2loss/SRqqt2/assignment_loss‚0.6922360982836994 1loss/SRqqt2/detection_loss‚0.6873232038004462 $ _timestamp‚1.784504620048476e+09 /loss/FRt1/assignment_loss‚2.454380111764618 1loss/FBt1/assignment_loss‚0.02813736886736004 &loss/total_loss‚10.161255586724565 epoch‚8£ ÃpÇi›‚(:Þ ¬ÂõÒ€—€À) disk./.usagePercent‚75.75010080093709 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚91.03125( disk.nvme0c0n1.out‚17618.39306640625 network.sent‚ 8.897165e+06 network.recv‚799853% disk./.usageGB‚1162.9534225463867 disk.dm-0.in‚ 101.51953125! disk.nvme1c1n1.in‚ 10.48828125% disk.nvme1c1n1.out‚4421.884765625* proc.memory.percent‚17.533445358276367 cpu‚26.492084552481348# disk.nvme1n1.out‚4421.884765625 disk.md0.in‚0 disk.nvme0n1.in‚91.03125' proc.memory.availableMB‚ 191.0078125" proc.memory.rssMB‚ 2872.6796875 proc.cpu.threads‚17 disk.nvme1n1.in‚ 10.48828125 disk.md0.out‚ 86.6953125# disk.dm-0.out‚19480.29345703125 disk.dm-1.out‚ 2375.30859375& disk.nvme0n1.out‚17618.39306640625$ memory_percent‚98.83418083190918^Åà›œ‚(: ¬ÂõÒáÒ«À gpu.0.gpu‚56.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚196.108' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.36933333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜b¨·j‚(:ß »ÂõÒ¹¦û¿% disk./.usageGB‚1162.9532737731934% disk.nvme1n1.out‚4422.76318359375 disk.md0.in‚0 disk.nvme0c0n1.in‚91.03125& disk.nvme0n1.out‚17627.24658203125# memory_percent‚98.8328218460083( proc.memory.availableMB‚ 191.23046875 proc.cpu.threads‚17 network.recv‚803902! disk.nvme1c1n1.in‚ 10.48828125 disk.nvme1n1.in‚ 10.48828125 disk.md0.out‚ 87.0234375 disk.nvme0n1.in‚91.03125" proc.memory.rssMB‚ 2872.6796875' disk.nvme1c1n1.out‚4422.76318359375 disk.dm-0.in‚ 101.51953125 disk.dm-0.out‚19488.89453125! disk.dm-1.out‚2375.7314453125* proc.memory.percent‚17.533445358276367 network.sent‚ 8.946744e+06) disk./.usagePercent‚75.75009111045064 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚17627.24658203125 cpu‚26.4890071905633_ñV ž‚(:… »ÂõÒ£´µÀ gpu.0.gpu‚54.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚142.977' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚47.659 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H«€2Ÿ- _wandbruntime‚2793  _runtime‚2793ArÜCÉ Ã ' lr-AdamW/pg1‚0.00010578907967828307 ' lr-AdamW/pg2‚0.00010578907967828307 trainer/global_step‚6799 % _timestamp‚1.7845046390072255e+09 _runtime‚2793.070383313 _step‚548|H ÿÎ¡È ' lr-AdamW/pg1‚0.00010578907967828307 ' lr-AdamW/pg2‚0.00010578907967828307 trainer/global_step‚6799 % _timestamp‚1.7845046390072255e+09 _runtime‚2793.070383313 _step‚548¤ éVŒ2¢- _wandbruntime‚2793  _runtime‚2793Uá s£ÒO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4918768884736 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68903620 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47361385 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7845046390082166e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47361137 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚261882024164352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4779168132096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743|7³ƒe!477248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚250146990756864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 trainer/global_step‚6799 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚258487222993168 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4779256154112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68893477 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 190899200 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51686531 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚266661175257088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚263251457010492 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215802880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51686533 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚255065950540800 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚263251560198326 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4779168132096 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚266661175257088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4764320165798 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68903620 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29926824 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚255065734737920v†É›Ý'¤×O L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚261882007124992 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68893477 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚261882007124992 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4779256154112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 120590153 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29926827 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚250146965853184 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 120580008 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚266661280318464 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚250146990756864 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 trainer/global_step‚6799 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚258487222993168 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4779256154112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚68893477 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 190899200 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚51686531 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚266661175257088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚263251457010492 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215802880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚51686533 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚255065950540800 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚263251560198326 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4779168132096 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚266661175257088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4764320165798 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚68903620 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29926824 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚255065734737920 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4918768884736 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68903620 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47361385 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7845046390082166e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47361137 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚261882024164352 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4779168132096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47368079 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120597227 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120607386 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29931114 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚250182661010432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4764996117354 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4779934270976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17436962 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 120607386 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47367815 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68903375 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚250182685914112 _step‚550 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚261919390531584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚255102339313152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚261919373492224 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚258524112100288 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚258524095060928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _runtime‚2793.337503777 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152¥‰‡§áM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68903375 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚255102127711744 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4779934270976 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _runtime‚2793.337503777 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 trainer/global_step‚6799 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47368079 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 120597227 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 120607386 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29931114 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚250182661010432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4764996117354 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4779934270976 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17436962 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 120607386 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47367815 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68903375 ;5DeviceStatsMonitor.on_train_bÀˆ‚Á`atch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚250182685914112 _step‚550 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚261919390531584 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚255102339313152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚261919373492224 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚258524112100288 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚258524095060928 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4953927852032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 121452955 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52052581 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30141324 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69390229 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30141327 trainer/global_step‚6849 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69400372 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚268563536421888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52052583 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 121442810 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47699794 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚256885676471296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69400372 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚265128993061734 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4813193895936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17558467 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17558223 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 _step‚553 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 _runtime‚ 2812.60160742 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚256885471154176 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69390229 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47699547 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚263750325486592 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚251931723715584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚265128889873900 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4813105873920 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4813105873920 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚260330843420528 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4798149641206 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚263750342525952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚263750325486592 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4798063492732 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7845046585385795e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚268563431360512 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚268563431360512 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4953747438592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚260330826381168 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0­†¯tÜ'²ÖO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52052581 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4813193895936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚251931748619264 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚263750342525952 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚268563536421888 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4953927852032 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 121452955 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52052581 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30141324 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69390229 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30141327 trainer/global_step‚6849 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69400372 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚268563536421888 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52052583 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 121442810 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚47699794 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚256885676471296 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69400372 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚244 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚265128993061734 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4813193895936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17558467 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17558223 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 _step‚553 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 _runtime‚ 2812.60160742 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚256885471154176 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69390229 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47699547 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚263750325486592 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚251931723715584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚265128889873900 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4813105873920 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4813105873920 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚260330843420528 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4798149641206 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚263750342525952 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚263750325486592 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4798063492732 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7845046585385795e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚268563431360512 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚268563431360512 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4953747438592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚260330826381168 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0©±Í<³2³- _wandbruntime‚2812  _runtime‚2812kP¬! ´ÜM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚265166541081050 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚263787691853824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17560645 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4813872012800 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚251967443776512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121470188 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4954440051200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 D>+è‚;ÁDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚265166437893104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚554 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47706259 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17560895 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4798739444176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4813783983616 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69410284 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚268601580905984 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30145614 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52059902 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52059904 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 _runtime‚2812.869738221 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69400127 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4813872012800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚251967418872832 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 121460029 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 121460029 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4954628846080 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4813872012800 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚251967443776512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 121470188 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4954440051200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚265166437893104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚554 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47706259 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17560895 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4798739444176 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4813783983616 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69410284 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚268601580905984 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30145614 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52059902 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52059904 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 _runtime‚2812.869738221 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69400127 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4813872012800 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚251967418872832 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 121460029 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 121460029 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4954628846080 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455ª-T—l2¶- _wandbruntime‚2812  _runtime‚2812-‰–· 2loss/FBt1/assignment_loss‚0.024942704137117744 epoch‚8 trainer/global_step‚6849 /loss/FRt1/assignment_loss‚2.535941383758175 0loss/FRt2/assignment_loss‚1.1321469607176604 2loss/SRqqt2/assignment_loss‚0.6219904695760535 /loss/FRt1/detection_loss‚0.5559421113161201 0loss/SRqqt1/detection_loss‚1.232690862236207 1loss/SRqqt2/detection_loss‚0.6656996468741647 %loss/total_loss‚9.882021879733886 _runtime‚2812.870457383 2loss/SRqqt1/assignment_loss‚0.6426442578312278 2loss/FBt2/assignment_loss‚0.022264443405592214 /loss/FRt2/detection_loss‚1.3880048168512682 /loss/FBt1/detection_loss‚0.5276583327481953 /loss/FBt2/detection_loss‚0.5320958902821038 % _timestamp‚1.7845046588074412e+09 _step‚555ü6 ¸† /loss/FBt2/detection_loss‚0.5320958902821038 % _timestamp‚1.7845046588074412e+09 _step‚555 2loss/FBt1/assignment_loss‚0.024942704137117744 epoch‚8 trainer/global_step‚6849 /loss/FRt1/assignment_loss‚2.535941383758175 0loss/FRt2/assignment_loss‚1.1321469607176604 2loss/SRqqt2/assignment_loss‚0.6219904695760535 /loss/FRt1/detection_loss‚0.5559421113161201 0loss/SRqqt1/detection_loss‚1.232690862236207 1loss/SRqqt2/detection_loss‚0.6656996468741647 %loss/total_loss‚9.882021879733886 _runtime‚2812.870457383 2loss/SRqqt1/assignment_loss‚0.6426442578312278 2loss/FBt2/assignment_loss‚0.022264443405592214 /loss/FRt2/detection_loss‚1.3880048168512682 /loss/FBt1/detection_loss‚0.5276583327481953«×þ<,e¹‚(:Ú ÙÂõÒ¹ÞÝ¿ network.recv‚811917% disk./.usageGB‚1163.0153884887695 disk.dm-1.out‚2376.072265625 disk.rbd0.in‚0 disk.md0.out‚ 87.5703125 cpu‚26.48295321790782 proc.cpu.threads‚17 network.sent‚ 9.013936e+06 disk.nvme0n1.in‚91.03125! disk.nvme1c1n1.in‚ 10.48828125 disk.nvme1n1.in‚ 10.48828125! disk.nvme1n1.out‚ 4423.9609375# disk.dm-0.out‚19521.44775390625' proc.memory.availableMB‚ 126.1796875* proc.memory.percent‚17.533445358276367 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚91.03125& disk.nvme0n1.out‚17660.12548828125# disk.nvme1c1n1.out‚ 4423.9609375 disk.md0.in‚0 disk.dm-0.in‚ 101.51953125" proc.memory.rssMB‚ 2872.6796875) disk./.usagePercent‚75.75413701278421( disk.nvme0c0n1.out‚17660.12548828125$ memory_percent‚99.22986030578613²Zš’º‚(:’ ÙÂõÒÝô²À gpu.0.gpu‚90.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚182.738' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚60.912666666666674 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0PÃ)Ä2»- _wandbruntime‚2832  _runtime‚2832’…— ɼà ' lr-AdamW/pg1‚0.00010558022891496727 ' lr-AdamW/pg2‚0.00010558022891496727 trainer/global_step‚6899 % _timestamp‚1.7845046780868995e+09 _runtime‚2832.151571792 _step‚556¦³‚ÐÎ½È ' lr-AdamW/pg1‚0.00010558022891496727 ' lr-AdamW/pg2‚0.00010558022891496727 trainer/global_step‚6899 % _timestamp‚1.7845046780868995e+09 _runtime‚2832.151571792 _step‚556¬ÌZ‚2¾- _wandbruntime‚2832  _runtime‚2832ti°JØ'¿ÒO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122315757 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69897124 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚258705389636096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚270465687463936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚265618660887552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52418631 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17682122 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚267006322737308 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚265618643848192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚258705182221824 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚267006425925142 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52418631 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48037946 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚262174429769168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚270465792525312 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52418633 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69886981 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚270465792525312 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚557 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30355827 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4988883154432 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69897124 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚270465687463936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚253716481577984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4831979116614 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚265618643848192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4847131637760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845046780875268e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4847043615744 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52418633 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17682385 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4847131637760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4847043615744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _runtime‚2832.151650802 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚265618660887552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69886981 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48038212 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 trainer/global_step‚6899 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30355824 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚253716506481664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330•ãˆšãÀ×O TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69897124 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚270465687463936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚253716481577984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4831979116614 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚265618643848192 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4847131637760 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7845046780875268e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4847043615744 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52418633 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17682385 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.alÎTÌÍúlocated‚ 4847131637760 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4847043615744 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _runtime‚2832.151650802 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚265618660887552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69886981 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48038212 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 trainer/global_step‚6899 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30355824 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚253716506481664 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 122315757 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69897124 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚258705389636096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚270465687463936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚265618660887552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52418631 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17682122 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚267006322737308 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚265618643848192 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚258705182221824 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚267006425925142 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52418631 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48037946 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚262174429769168 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚270465792525312 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52418633 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚69886981 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚270465792525312 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚557 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30355827 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4988883154432­ØïÒÝ2Á- _wandbruntime‚2832  _runtime‚2832^iìÈâ&ÂÜM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 _runtime‚2832.417543649 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52425954 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48044708 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122332990 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52425954 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4847721725440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚253752176735232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4847809754624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122322831 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122322831 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52425952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4989582022656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4847809754624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17684858 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17684594 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚267043973944458 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚258741574157312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69907036 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚262211301836928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4989397422080 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚262211318876288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚270503837009408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122332990 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4832655068170 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6899 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48044975 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4832568919584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69896879 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚270503837009408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚270503731940864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69907036ѨÎ%ç&ÃáM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69896879 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48044975 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4832568919584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69896879 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚270503837009408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚270503731940864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69907036 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 _runtime‚2832.417543649 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52425954 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48044708 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122332990 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52425954 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4847721725440 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚253752176735232 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4847809754624 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122322831 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122322831 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52425952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4989582022656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4847809754624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17684858 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17684594 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚267043973944458 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚258741574157312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69907036 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚262211301836928 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4989397422080 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚262211318876288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚270503837009408 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122332990 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4832655068170 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚264 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6899 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272367943567360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚267486979249152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30570327 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17806223 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4880981357568 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 123168414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚272368048628736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17805951 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚260525159217664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚6949 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48376550 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5023710265856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52784683 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4881069379584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚272367943567360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4880981357568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚561 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52784683 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70393876 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70383733 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚260524949706240 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚268883755600716 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚255501264344064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70383733 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4865808592022 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚255501239440384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚272368048628736 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52784681 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123178559 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚268883858788550 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 $ _timestamp‚1.784504697321868e+09 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5023894873600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70393876 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4881069379584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52784681 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚264018050196528ßò‹Ü'ÐÖO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4880981357568 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 123168414 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚272368048628736 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17805951 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚260525159217664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚6949 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48376550 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5023710265856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52784683 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4881069379584 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚272367943567360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4880981357568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚561 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52784683 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70393876 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚70383733 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚272 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚260524949706240 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚268883755600716 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚255501264344064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70383733 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4865808592022 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚255501239440384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚272368048628736 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52784681 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123178559 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚268883858788550 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 $ _timestamp‚1.784504697321868e+09 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5023894873600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70393876 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4881069379584 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52784681 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚264018050196528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚267486962209792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4865722443548 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264018033157168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚267486979249152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚275 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272367943567360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚267486979249152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30570327 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17806223 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104±•ªÓ2Ñ- _wandbruntime‚2851  _runtime‚2851¸’â&ÒÜM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70393631 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 % _timestamp‚1.7845046979001198e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52792004 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52792002 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30574614 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚272405988044288 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48383233 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30574617 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4866398394992 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52792002 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚267524345616384 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚272406093112832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17808616 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚260561551211520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4881659467264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4881659467264 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚255536934597632 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚268921406807866 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚272406093112832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70403788 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚267524345616384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52792004 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚264054922264288 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17808373 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚267524328577024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚2851.963428573 _step‚562 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5024409206784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123185633 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚267524328577024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4881747496448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4881747496448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123185633 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚255536959501312 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70403788 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4866484543578 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123195792 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚264054905224928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 trainer/global_step‚6949 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123195792 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48382987 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ÞâqÓáM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚272406093112832 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17808616 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚260561551211520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4881659467264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4881659467264 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚255536934597632 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚268921406807866 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚272406093112832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70403788 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚267524345616384 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52792004 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_ba;g+övtch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚264054922264288 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17808373 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚267524328577024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚2851.963428573 _step‚562 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5024409206784 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123185633 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚267524328577024 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4881747496448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4881747496448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 123185633 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚255536959501312 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70403788 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4866484543578 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 123195792 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚264054905224928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 trainer/global_step‚6949 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123195792 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48382987 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70393631 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 % _timestamp‚1.7845046979001198e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52792004 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52792002 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30574614 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚272405988044288 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚48383233 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30574617 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4866398394992 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52792002 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚267524345616384 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0²  2Ô- _wandbruntime‚2851  _runtime‚2851æÕ$1Õ 2loss/SRqqt1/assignment_loss‚0.6642127953228586 2loss/SRqqt2/assignment_loss‚0.6666962725983467 1loss/FBt1/assignment_loss‚0.07259397732386029 /loss/FRt1/detection_loss‚0.5596011906079421 .loss/FRt2/detection_loss‚1.591047137124525 /loss/FBt1/detection_loss‚0.5294745478158196 _step‚563 0loss/FRt1/assignment_loss‚2.4998590923418664 1loss/SRqqt2/detection_loss‚0.6407414225588686 % _timestamp‚1.7845046979010859e+09 _runtime‚2851.963936963 2loss/FBt2/assignment_loss‚0.012678929380994575 epoch‚8 trainer/global_step‚6949 /loss/FRt2/assignment_loss‚1.151394641301671 1loss/SRqqt1/detection_loss‚1.1743344032309122 /loss/FBt2/detection_loss‚0.5512594340971726 &loss/total_loss‚10.113893843704837‘Ô†ú Ö† _runtime‚2851.963936963 2loss/FBt2/assignment_loss‚0.012678929380994575 epoch‚8 trainer/global_step‚6949 /loss/FRt2/assignment_loss‚1.151394641301671 1loss/SRqqt1/detection_loss‚1.1743344032309122 /loss/FBt2/detection_loss‚0.5512594340971726 &loss/total_loss‚10.113893843704837 2loss/SRqqt1/assignment_loss‚0.6642127953228586 2loss/SRqqt2/assignment_loss‚0.6666962725983467 1loss/FBt1/assignment_loss‚0.07259397732386029 /loss/FRt1/detection_loss‚0.5596011906079421 .loss/FRt2/detection_loss‚1.591047137124525 /loss/FBt1/detection_loss‚0.5294745478158196 _step‚563 0loss/FRt1/assignment_loss‚2.4998590923418664 1loss/SRqqt2/detection_loss‚0.6407414225588686 % _timestamp‚1.7845046979010859e+09³¥ÎËjׂ(:ß †ÃõÒ¤ŸžÀ$ disk.nvme1n1.out‚4426.4248046875 disk.dm-0.in‚ 101.51953125( proc.memory.availableMB‚ 125.86328125" proc.memory.rssMB‚ 2872.6796875 cpu‚26.47391868411501 network.sent‚ 9.15726e+06 network.recv‚823871% disk./.usageGB‚1162.9604301452637 disk.nvme0c0n1.in‚91.03125# disk.dm-0.out‚19533.49755859375 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚17672.76806640625 disk.md0.in‚0 disk.dm-1.out‚2376.373046875* proc.memory.percent‚17.533445358276367( disk./.usagePercent‚75.7505572476967 disk.nvme0n1.in‚91.03125& disk.nvme1c1n1.out‚4426.4248046875 disk.nvme1n1.in‚ 10.48828125 disk.md0.out‚ 88.8671875$ memory_percent‚99.23179149627686 proc.cpu.threads‚17( disk.nvme0c0n1.out‚17672.76806640625! disk.nvme1c1n1.in‚ 10.48828125ÍS¦ ›Ø‚(: †ÃõÒ•»ÄÀ gpu.0.gpu‚69.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚113.351' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.78366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤ˆß¶2Ù- _wandbruntime‚2870  _runtime‚2870šyÊÉÚà trainer/global_step‚6999 % _timestamp‚1.7845047168698862e+09 _runtime‚2870.933034955 _step‚564 ' lr-AdamW/pg1‚0.00010536820572119714 ' lr-AdamW/pg2‚0.00010536820572119714RÑ‚rÎÛÈ ' lr-AdamW/pg2‚0.00010536820572119714 trainer/global_step‚6999 % _timestamp‚1.7845047168698862e+09 _runtime‚2870.933034955 _step‚564 ' lr-AdamW/pg1‚0.00010536820572119714´8ti2Ü- _wandbruntime‚2870  _runtime‚2870Õâ:Ø'ÝÒO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚274270199670784 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4914919099392 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17929300 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚269355280571392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚269355280571392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚265861653584528 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚262344655817728 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚269355297610752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚257285997302784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚2870.933936046 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30784827 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚265861636545168 % _timestamp‚1.7845047168708324e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4914919099392 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124031216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48714127 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124041361 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53150731 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚274270304732160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30784824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17929059 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48713883 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚274270199670784 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70880485 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53150733 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚265861653584528 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚262344655817728 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚269355297610752 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚257285997302784 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚2870.933936046 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30784827 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚265861636545168 % _timestamp‚1.7845047168708324e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4914919099392 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚244 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124031216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚48714127 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124041361 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53150731 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚274270304732160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30784824 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17929059 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚48713883 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚274270199670784 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚70880485 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53150733 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚274270199670784 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4914919099392 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚17929300 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚269355280571392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚269355280571392 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0µœm12ß- _wandbruntime‚2871  _runtime‚2871DëQ üàÛM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚262381261095424 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4915597209088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 REDeviceStatsMonitor.on_train_batch_end/inactive_sõÉÎ|å"plit.small_pool.freed‚17931471 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4900314018986 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4900227870400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 _step‚566 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚269392646938624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚269392663977984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚269392663977984 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚257321692460032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30789114 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5059357033984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30789117 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚265898508612928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4915685238272 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚269392646938624 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4915685238272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124058594 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53158054 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70900540 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53158052 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70900540 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48720585 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚270798736483328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚274308244147712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚270798839671274 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70890383 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70890383 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚257321717363712 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586‘Ê¿Ðæ&áàM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚265898508612928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4915685238272 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚269392646938624 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4915685238272 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124058594 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53158054 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚70900540 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53158052 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚70900540 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚48720585 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚270798736483328 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚274308244147712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚270798839671274 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚70890383 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚70890383 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚257321717363712 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚262381261095424 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4915597209088 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17931471 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4900314018986 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4900227870400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 _step‚566 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚269392646938624 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚269392663977984 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚269392663977984 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚257321692460032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30789114 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5059357033984 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30789117 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728¶Øûp=2â- _wandbruntime‚2871  _runtime‚2871î=2 ã‚ trainer/global_step‚6999 _runtime‚2871.200715631 0loss/FRt1/assignment_loss‚2.5349088259138917 0loss/FRt2/assignment_loss‚1.0740711431877286 2loss/FBt2/assignment_loss‚0.012867032605849586 .loss/FBt1/detection_loss‚0.918213477008976 epoch‚8 % _timestamp‚1.7845047171378849e+09 /loss/FBt2/detection_loss‚0.4891338486362904 2loss/SRqqt2/assignment_loss‚0.6535356958421904 /loss/FRt1/detection_loss‚0.5242216928514633 1loss/SRqqt2/detection_loss‚0.6008283148337206 &loss/total_loss‚10.185045961903981 _step‚567 2loss/SRqqt1/assignment_loss‚0.8046081274369499 2loss/FBt1/assignment_loss‚0.008121223425490598 /loss/FRt2/detection_loss‚1.3122272245567657 0loss/SRqqt1/detection_loss‚1.252309355604666äÝ ä‡ _step‚567 2loss/SRqqt1/assignment_loss‚0.8046081274369499 2loss/FBt1/assignment_loss‚0.008121223425490598 /loss/FRt2/detection_loss‚1.3122272245567657 0loss/SRqqt1/detection_loss‚1.252309355604666 trainer/global_step‚6999 _runtime‚2871.200715631 0loss/FRt1/assignment_loss‚2.5349088259138917 0loss/FRt2/assignment_loss‚1.0740711431877286 2loss/FBt2/assignment_loss‚0.012867032605849586 .loss/FBt1/detection_loss‚0.918213477008976 epoch‚8 % _timestamp‚1.7845047171378849e+09 /loss/FBt2/detection_loss‚0.4891338486362904 2loss/SRqqt2/assignment_loss‚0.6535356958421904 /loss/FRt1/detection_loss‚0.5242216928514633 1loss/SRqqt2/detection_loss‚0.6008283148337206 &loss/total_loss‚10.185045961903981·Ò¬°[lå‚(:á •ÃõÒ‹…ø¿ network.recv‚827997( disk./.usagePercent‚75.7516343825377 disk.nvme1n1.in‚ 10.48828125 disk.md0.in‚0 disk.md0.out‚ 89.5546875! disk.nvme0c0n1.in‚ 91.05859375 network.sent‚ 9.218458e+06% disk./.usageGB‚1162.9769668579102" disk.dm-0.out‚19542.9033203125 disk.rbd0.in‚0 disk.nvme0n1.in‚ 91.05859375! disk.nvme1c1n1.in‚ 10.48828125$ memory_percent‚99.23200607299805" proc.memory.rssMB‚ 2872.6796875$ disk.nvme1n1.out‚4965.5302734375 disk.dm-0.in‚ 101.546875 disk.dm-1.in‚0& disk.nvme1c1n1.out‚4965.5302734375& proc.memory.availableMB‚ 125.828125* proc.memory.percent‚17.533445358276367 cpu‚26.470949767397194 disk.rbd0.out‚0' disk.nvme0c0n1.out‚17679.7275390625% disk.nvme0n1.out‚17679.7275390625 proc.cpu.threads‚17! disk.dm-1.out‚2911.5888671875þ¯ÿœæ‚(:‘ •ÃõÒÒÇ£À gpu.0.gpu‚90.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.841' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.28033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªH÷½2ç- _wandbruntime‚2890  _runtime‚2890ÒÎÉèà trainer/global_step‚7049 % _timestamp‚1.7845047364177978e+09 _runtime‚2890.480931262 _step‚568 ' lr-AdamW/pg1‚0.00010526100869993619 ' lr-AdamW/pg2‚0.00010526100869993619˜†U#ÎéÈ _runtime‚2890.480931262 _step‚568 ' lr-AdamW/pg1‚0.00010526100869993619 ' lr-AdamW/pg2‚0.00010526100869993619 trainer/global_step‚7049 % _timestamp‚1.7845047364177978e+09¸(¼»Ã2ê- _wandbruntime‚2890  _runtime‚2890|xBØ'ëÒO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71377237 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53516783 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18053061 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚7049 % _timestamp‚1.7845047364187024e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49052643 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚271223615972352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚276172455774208 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚271223598932992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚271223615972352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53516781 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30999324 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4948856841216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124904163 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30999327 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚259070780068864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4948856841216 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71387380 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚271223598932992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚264164390595072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚276172455774208 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚569 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚267705239933168 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚264164598009344 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53516783 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71387380 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18053316 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚272638621327532 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53516781 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4948944863232 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚267705256972528 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4933381394364 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124894018 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0¡Z¸ì×O ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4948944863232 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚cèf=%'267705256972528 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4933381394364 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 124894018 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71377237 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53516783 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18053061 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚7049 % _timestamp‚1.7845047364187024e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49052643 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚271223615972352 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚276172455774208 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚271223598932992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚271223615972352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53516781 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30999324 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4948856841216 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 124904163 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30999327 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚259070780068864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4948856841216 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71387380 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚271223598932992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚264164390595072 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚276172455774208 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚569 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚267705239933168 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚264164598009344 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53516783 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71387380 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18053316 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚272638621327532 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53516781 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592¹Ó}ÑÙ2í- _wandbruntime‚2890  _runtime‚2890Á{€¹á&îÛM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚264200782837760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚271260982339584 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124921396 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚264200992342016 _runtime‚2890.748453981 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚271260982339584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚271260965300224 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18055782 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚272676272534682 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124921396 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71397292 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49059133 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124911237 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4949534950912 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71387135 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49059399 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚7049 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53524104 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚267742112000928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 $ _timestamp‚1.784504736685176e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4949534950912 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4934057345808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚271260965300224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚276210605319680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4934143494394 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71387135 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31003617 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚276210500251136 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 124911237 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚276210605319680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4934143494394 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71387135 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31003617 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚276210500251136 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 124911237 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 124921396 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚264200992342016 _runtime‚2890.748453981 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚271260982339584 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚271260965300224 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18055782 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚272676272534682 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 124921396 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71397292 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49059133 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 124911237 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4949534950912 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71387135 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49059399 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚7049 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53524104 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚267742112000928 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455º,U"?2ð- _wandbruntime‚2890  _runtime‚2890²»ñý trainer/global_step‚7049 1loss/SRqqt1/assignment_loss‚0.710797148292486 1loss/FBt1/assignment_loss‚0.04356588859374256 /loss/FRt1/detection_loss‚0.5640067097667277 %loss/total_loss‚9.579296608957634 1loss/FBt2/assignment_loss‚0.07299862354992274 epoch‚8 _runtime‚2890.749676403 _step‚571 0loss/FRt2/assignment_loss‚0.9216520014538694 /loss/FBt1/detection_loss‚0.6749412343651057 $ _timestamp‚1.784504736686578e+09 /loss/FRt1/assignment_loss‚2.436741318538099 2loss/SRqqt2/assignment_loss‚0.5581366538556347 /loss/FRt2/detection_loss‚1.3810939970078753 1loss/SRqqt1/detection_loss‚1.0194508787713148 0loss/SRqqt2/detection_loss‚0.720678966237328 /loss/FBt2/detection_loss‚0.4752331885255282ÿ†ûáò‚ /loss/FRt1/assignment_loss‚2.436741318538099 2loss/SRqqt2/assignment_loss‚0.5581366538556347 /loss/FRt2/detection_loss‚1.3810939970078753 1loss/SRqqt1/detection_loss‚1.0194508787713148 0loss/SRqqt2/detection_loss‚0.720678966237328 /loss/FBt2/detection_loss‚0.4752331885255282 trainer/global_step‚7049 1loss/SRqqt1/assignment_loss‚0.710797148292486 1loss/FBt1/assignment_loss‚0.04356588859374256 /loss/FRt1/detection_loss‚0.5640067097667277 %loss/total_loss‚9.579296608957634 1loss/FBt2/assignment_loss‚0.07299862354992274 epoch‚8 _runtime‚2890.749676403 _step‚571 0loss/FRt2/assignment_loss‚0.9216520014538694 /loss/FBt1/detection_loss‚0.6749412343651057 $ _timestamp‚1.784504736686578e+09» ÈÃoó‚(:ä ¤ÃõÒ‡±ç¿ disk.dm-0.in‚ 104.765625! disk.dm-1.out‚2931.9091796875 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 94.27734375( disk.nvme0c0n1.out‚17720.35986328125) disk./.usagePercent‚75.87369003507102% disk./.usageGB‚1164.8508262634277! disk.nvme1c1n1.in‚ 10.48828125 disk.nvme1n1.in‚ 10.48828125 disk.dm-0.out‚21334.35546875$ memory_percent‚99.22544956207275( proc.memory.availableMB‚ 126.90234375 cpu‚26.46809421015877 network.sent‚ 9.267865e+06 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 94.27734375 network.recv‚832627% disk.nvme1n1.out‚6742.92138671875& disk.nvme0n1.out‚17720.35986328125" proc.memory.rssMB‚ 2872.6796875* proc.memory.percent‚17.533445358276367 proc.cpu.threads‚17' disk.nvme1c1n1.out‚6742.92138671875 disk.md0.in‚0 disk.md0.out‚ 92.640625ž÷ $÷ô‚(: ¤ÃõÒŠüÁÀ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.557' gpu.0.enforcedPowerLimitWatts‚300.0( ›Ž£æ£gpu.0.powerPercent‚37.18566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0kyœírõ‚(:ç ³ÃõÒ¦¶¿" proc.memory.rssMB‚ 2872.6796875 network.sent‚ 9.33157e+06% disk./.usageGB‚1165.6843910217285 disk.md0.in‚0 disk.dm-0.in‚ 109.21875' disk.nvme1c1n1.out‚7714.71142578125% disk.nvme1n1.out‚7714.71142578125 network.recv‚837166 disk.dm-1.in‚0" disk.dm-1.out‚2932.76318359375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 98.73046875! disk.nvme1c1n1.in‚ 10.48828125* proc.memory.percent‚17.533445358276367 cpu‚26.46518726862144# disk.dm-0.out‚22308.74365234375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚17724.94189453125 proc.cpu.threads‚17) disk./.usagePercent‚75.92798508527787 disk.md0.out‚ 93.1640625! disk.nvme0c0n1.in‚ 98.73046875& disk.nvme0n1.out‚17724.94189453125 disk.nvme1n1.in‚ 10.48828125$ memory_percent‚99.22716617584229( proc.memory.availableMB‚ 126.62109375)ÏãØšö‚(: ³ÃõÒà›ò¿ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚185.621' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚61.87366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0^‰Þ2÷- _wandbruntime‚2910  _runtime‚2910%†ÍÀÉøà ' lr-AdamW/pg1‚0.00010515302368084552 ' lr-AdamW/pg2‚0.00010515302368084552 trainer/global_step‚7099 % _timestamp‚1.7845047560603864e+09 _runtime‚2910.124560579 _step‚572m†MWÎùÈ _runtime‚2910.124560579 _step‚572 ' lr-AdamW/pg1‚0.00010515302368084552 ' lr-AdamW/pg2‚0.00010515302368084552 trainer/global_step‚7099 % _timestamp‚1.7845047560603864e+09¼ÜíÅ 2ú- _wandbruntime‚2910  _runtime‚2910Þ‡6Ø'ûÒO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4967297018246 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53882833 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚274516054190940 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4967210869772 _step‚573 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49391066 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5128770375680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4982794583040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚278074816939008 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚278074711877632 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚273091917294592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚260855513027584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71884132 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49390824 trainer/global_step‚7099 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145ÔÅ¿Ý'ü×O ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4967297018246 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53882833 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚274516054190940 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4967210869772 _step‚573 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49391066 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5128770375680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4982794583040 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚278074816939008 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚278074711877632 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚273091917294592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚260855513027584 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71884132 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49390824 trainer/global_step‚7099 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680½',¯º2ý- _wandbruntime‚2910  _runtime‚2910Õ|]râ&þÜM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4983560721920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚269585715388928 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845047563253417e+09 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125774039 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚278112861423104 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4967972969802 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚278112756354560 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125784198 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49397758 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚266020501802496 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71883887 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚278112861423104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚273129300701184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚260891233088512 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125784198 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71894044 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18179641 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4967886821216 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4983560721920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚7099 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚273129283661824 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53890152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5129476121088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53890152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚273129300701184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4983472692736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31218117 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚269585732428288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4983472692736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71883887 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49397501 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152q à9ÿáM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚273129300701184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all"Po®%.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4983472692736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31218117 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚269585732428288 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4983472692736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71883887 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49397501 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4983560721920 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚269585715388928 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7845047563253417e+09 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125774039 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚278112861423104 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4967972969802 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚278112756354560 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125784198 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49397758 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚266020501802496 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚71883887 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚278112861423104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚273129300701184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚260891233088512 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125784198 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71894044 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18179641 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4967886821216 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4983560721920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚7099 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚273129283661824 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53890152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5129476121088 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53890152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0¾ˆ-„2€- _wandbruntime‚2910  _runtime‚2910…©¡k &loss/total_loss‚10.006774747299588 epoch‚8 trainer/global_step‚7099 _runtime‚2910.389144084 _step‚575 2loss/SRqqt2/assignment_loss‚0.5143289579409946 0loss/FRt1/assignment_loss‚2.4661164236791207 0loss/FRt2/assignment_loss‚1.0888809433170394 1loss/FBt1/assignment_loss‚0.09864533340396892 /loss/FRt2/detection_loss‚1.4481333743187361 1loss/SRqqt1/detection_loss‚1.0560464471495017 1loss/SRqqt2/detection_loss‚0.7203630419002355 .loss/FBt1/detection_loss‚0.766207720552172 /loss/FBt2/detection_loss‚0.6481838935525854 2loss/SRqqt1/assignment_loss‚0.6551829422794234 2loss/FBt2/assignment_loss‚0.007672405563385887 .loss/FRt1/detection_loss‚0.537013263642424 % _timestamp‚1.7845047563263233e+09ki1k ‚† 1loss/FBt1/assignment_loss‚0.09864533340396892 /loss/FRt2/detection_loss‚1.4481333743187361 1loss/SRqqt1/detection_loss‚1.0560464471495017 1loss/SRqqt2/detection_loss‚0.7203630419002355 .loss/FBt1/detection_loss‚0.766207720552172 /loss/FBt2/detection_loss‚0.6481838935525854 2loss/SRqqt1/assignment_loss‚0.6551829422794234 2loss/FBt2/assignment_loss‚0.007672405563385887 .loss/FRt1/detection_loss‚0.537013263642424 % _timestamp‚1.7845047563263233e+09 &loss/total_loss‚10.006774747299588 epoch‚8 trainer/global_step‚7099 _runtime‚2910.389144084 _step‚575 2loss/SRqqt2/assignment_loss‚0.5143289579409946 0loss/FRt1/assignment_loss‚2.4661164236791207 0loss/FRt2/assignment_loss‚1.0888809433170394¿¼íUïrƒ‚(:ç ÂÃõÒœ‚õ¿ disk.dm-0.in‚ 110.48046875# disk.dm-0.out‚22328.88427734375( disk.nvme0c0n1.out‚17737.89892578125& disk.nvme0n1.out‚17737.89892578125' disk.nvme1c1n1.out‚7723.84814453125$ memory_percent‚99.22585487365723 network.recv‚840811% disk.nvme1n1.out‚7723.84814453125 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 99.9921875' proc.memory.availableMB‚ 126.8359375 cpu‚26.462463558848572 proc.cpu.threads‚17% disk./.usageGB‚1165.6842041015625 disk.md0.out‚ 94.046875 disk.nvme0c0n1.in‚ 99.9921875 network.sent‚ 9.349277e+06 disk.md0.in‚0" disk.dm-1.out‚2932.86474609375 disk.rbd0.out‚0! disk.nvme1c1n1.in‚ 10.48828125" proc.memory.rssMB‚ 2872.6796875* proc.memory.percent‚17.533445358276367( disk./.usagePercent‚75.9279729100513 disk.nvme1n1.in‚ 10.48828125…Y㛄‚(: ÂÃõÒ〡À gpu.0.gpu‚36.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.49' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.163333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0d0wŠ2…- _wandbruntime‚2929  _runtime‚2929z-džÁ trainer/global_step‚7149 % _timestamp‚1.7845047756079338e+09 _runtime‚2929.671066916 _step‚576 & lr-AdamW/pg1‚0.0001050442523935244 & lr-AdamW/pg2‚0.0001050442523935244xYMóÌ‡Æ % _timestamp‚1.7845047756079338e+09 _runtime‚2929.671066916 _step‚576 & lr-AdamW/pg1‚0.0001050442523935244 & lr-AdamW/pg2‚0.0001050442523935244 trainer/global_step‚7149ÀæÄ;ô2ˆ- _wandbruntime‚2929  _runtime‚2929­q‹kØ'‰ÒO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚279977073042432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5001126493654 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72370741 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚267803824355328 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚262640270889984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54248881 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚274960235656192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54248881 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72380884 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279976967981056 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚262640295793664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49728009 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 % _timestamp‚1.7845047756089149e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚267804044352512 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 126629767 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 195093504 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚276393487054348 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚271392463748528 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49727740 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54248883 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5001040345180 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚279976967981056 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18299682 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5016820346880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 trainer/global_step‚7149 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126619622 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72380884 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5016732324864 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31428327 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚2929.672053917 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚269 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚279977073042432 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5001126493654 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72370741 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚267803824355328 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚262640270889984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54248881 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚274960235656192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54248881 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚266 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚72380884 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279976967981056 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚262640295793664 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49728009 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 % _timestamp‚1.7845047756089149e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚267804044352512 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 126629767 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 195093504 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚276393487054348 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚271392463748528 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚49727740 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54248883 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5001040345180 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚279976967981056 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18299682 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_poY³Ã;ol.allocated‚ 5016820346880 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 trainer/global_step‚7149 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126619622 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72380884 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712Á’³¢¾2‹- _wandbruntime‚2929  _runtime‚2929CÅeâ&ŒÜM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚274997602023424 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49734753 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚280015012457984 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚262675990950912 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚271429318776928 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274997619062784 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _step‚578 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5164257139200 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚276431035073552 _runtime‚2929.938995392 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72390796 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31432614 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5017410434560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚271429335816288 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126647000 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚267840428496384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49734494 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54256202 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5017410434560 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54256204 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 126636841 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72380639 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18301880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚280015117526528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126647000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5017498463744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31432617 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18302136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚267840223186432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5164437545472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 % _timestamp‚1.7845047758756797e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72380639 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54256204 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5001716296624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5017498463744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5001802445210 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104×’Þ÷ç&áM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5001716296624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5017498463744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5001802445210 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚274997602023424 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49734753 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚280015012457984 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚262675990950912 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚271429318776928 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274997619062784 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _step‚578 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5164257139200 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚276431035073552 _runtime‚2929.938995392 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72390796 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31432614 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5017410434560 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚271429335816288 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 126647000 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚267840428496384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49734494 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54256202 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5017410434560 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54256204 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 126636841 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72380639 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18301880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚280015117526528 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 126647000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5017498463744 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31432617 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18302136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚267840223186432 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5164437545472 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 % _timestamp‚1.7845047758756797e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72380639 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚54256204 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448Â* a2Ž- _wandbruntime‚2929  _runtime‚2929^%a epoch‚8 1loss/FBt2/assignment_loss‚0.00507428944263197 1loss/SRqqt1/detection_loss‚0.9517642919681151 /loss/FBt1/detection_loss‚1.1542815240258775 trainer/global_step‚7149 _step‚579 /loss/FRt2/detection_loss‚1.4634305025958192 /loss/FBt2/detection_loss‚0.6196056403165969 % _timestamp‚1.7845047758767033e+09 _runtime‚2929.939917694 /loss/FRt1/assignment_loss‚2.458668052919151 0loss/FRt2/assignment_loss‚1.3298891387819274 1loss/SRqqt2/detection_loss‚0.7224957229770667 2loss/SRqqt1/assignment_loss‚0.6810819107977959 2loss/SRqqt2/assignment_loss‚0.6311276708018525 1loss/FBt1/assignment_loss‚0.06276036461889073 /loss/FRt1/detection_loss‚0.5534749427770717 &loss/total_loss‚10.633654052022797õÖCª † /loss/FBt2/detection_loss‚0.6196056403165969 % _timestamp‚1.7845047758767033e+09 _runtime‚2929.939917694 /loss/FRt1/assignment_loss‚2.458668052919151 0loss/FRt2/assignment_loss‚1.3298891387819274 1loss/SRqqt2/detection_loss‚0.7224957229770667 2loss/SRqqt1/assignment_loss‚0.6810819107977959 2loss/SRqqt2/assignment_loss‚0.6311276708018525 1loss/FBt1/assignment_loss‚0.06276036461889073 /loss/FRt1/detection_loss‚0.5534749427770717 &loss/total_loss‚10.633654052022797 epoch‚8 1loss/FBt2/assignment_loss‚0.00507428944263197 1loss/SRqqt1/detection_loss‚0.9517642919681151 /loss/FBt1/detection_loss‚1.1542815240258775 trainer/global_step‚7149 _step‚579 /loss/FRt2/detection_loss‚1.4634305025958192ÃâÜɲo‘‚(:ä ÑÃõÒ鲃À$ memory_percent‚99.22609329223633" proc.memory.rssMB‚ 2872.6796875* proc.memory.percent‚17.533445358276367( disk./.usagePercent‚75.9279351420015$ disk.nvme1n1.out‚7724.5341796875 disk.md0.out‚ 94.3828125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 99.9921875( disk.nvme0c0n1.out‚17744.83154296875! disk.nvme1c1n1.in‚ 10.48828125 network.sent‚ 9.409619e+06 disk.md0.in‚0 disk.dm-1.in‚0" disk.dm-1.out‚2933.04931640625& proc.memory.availableMB‚ 126.796875 cpu‚26.459520793353953 proc.cpu.threads‚17 network.recv‚843976 disk.nvme1n1.in‚ 10.48828125" disk.dm-0.out‚22335.5732421875 disk.nvme0n1.in‚ 99.9921875& disk.nvme0n1.out‚17744.83154296875% disk./.usageGB‚1165.6836242675781 disk.dm-0.in‚ 110.48046875& disk.nvme1c1n1.out‚7724.5341796875Rµšø’‚(:„ ÑÃõÒôî°À gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚152.385' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚50.795 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a«ý52“- _wandbruntime‚2948  _runtime‚2948ñ ÔÐɔà ' lr-AdamW/pg1‚0.00010493469658016575 ' lr-AdamW/pg2‚0.00010493469658016575 trainer/global_step‚7199 % _timestamp‚1.7845047948858175e+09 _runtime‚2948.948941569 _step‚580tUÎ•È ' lr-AdamW/pg1‚0.00010493469658016575 ' lr-AdamW/pg2‚0.00010493469658016575 trainer/global_step‚7199 % _timestamp‚1.7845047948858175e+09 _runtime‚2948.948941569 _step‚580Äý2Vê2–- _wandbruntime‚2948  _runtime‚2948t©èüe!—ÐO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5034869820588 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚278270919917756 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚276828554017792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚264425028752384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5050670066688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 $ _timestamp‚1.784504794886667e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚281879224084480 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72877636 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚281879224084480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31642824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚273236050097168 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54614931 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5050758088704 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚281879329145856 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚276828571057152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚273236067136528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5198715585024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5198528880128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18423353 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5050670066688 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50066180 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚278271023105590 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚281879329145856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚264425053656064 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚7199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127482424 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127492569 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batΨ2­qch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54614933 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72867493 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50065925 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269623769241088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚269623557632512 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _step‚581 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚276828571057152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31642827 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928‰Â¡Û'˜ÕO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31642824 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚273236050097168 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54614931 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5050758088704 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚281879329145856 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚276828571057152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚273236067136528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5198715585024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5198528880128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18423353 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5050670066688 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50066180 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚278271023105590 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚281879329145856 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚264425053656064 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚7199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 127482424 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 127492569 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54614933 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚72867493 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50065925 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269623769241088 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚269623557632512 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _step‚581 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚276828571057152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31642827 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5034869820588 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚278270919917756 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚276828554017792 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚264425028752384 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5050670066688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 $ _timestamp‚1.784504794886667e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚281879224084480 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72877636 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚281879224084480 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016ÅÈ•»C2™- _wandbruntime‚2949  _runtime‚2949ó â&šÜM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚281917373629952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31647117 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚2949.217329111 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚276865937424384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18425787 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚269659954053120 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5199412646912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚269660161460224 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54622252 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5051348176384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚273272922164928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5051348176384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚276865937424384 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72887548 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50072653 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31647114 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5199230143488 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚281917268561408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127499643 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127509802 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54622252 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72877391 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚276865920385024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚281917373629952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5051436205568 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72887548 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚264460748813312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54622254 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚273272939204288 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 % _timestamp‚1.7845047951540437e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72877391 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50072904 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚278308467936960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ÷å>ç&›áM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚264460748813312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54622254 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚273272939204288 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 % _timestamp‚1.7845047951540437e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72877391 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50072904 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚278308467936960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚281917373629952 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31647117 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚2949.217329111 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _step‚582 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚276865937424384 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18425787 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚269659954053120 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5199412646912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚269660161460224 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚54622252 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5051348176384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚273272922164928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5051348176384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚276865937424384 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚72887548 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50072653 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31647114 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5199230143488 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚281917268561408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127499643 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127509802 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54622252 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚72877391 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚276865920385024 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚281917373629952 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5051436205568 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72887548ÆT œ2œ- _wandbruntime‚2949  _runtime‚29493>HÍ ƒ 0loss/FRt2/assignment_loss‚1.1102469081717325 2loss/SRqqt2/assignment_loss‚0.6803049519671318 1loss/SRqqt1/detection_loss‚1.0150366400411686 1loss/SRqqt2/detection_loss‚0.6915632072017285 epoch‚8 trainer/global_step‚7199 % _timestamp‚1.7845047951550252e+09 _runtime‚2949.217846922 0loss/FBt1/assignment_loss‚0.2388497843948926 3loss/FBt2/assignment_loss‚0.0005066364429774458 /loss/FBt1/detection_loss‚0.6279778180146302 /loss/FBt2/detection_loss‚0.5926127650775015 _step‚583 0loss/FRt1/assignment_loss‚2.2886836172405043 &loss/total_loss‚10.000434800472677 2loss/SRqqt1/assignment_loss‚0.6529892462410576 /loss/FRt1/detection_loss‚0.5888214383185474 /loss/FRt2/detection_loss‚1.5128417873608047q^Fžˆ 2loss/SRqqt1/assignment_loss‚0.6529892462410576 /loss/FRt¹è¶cÈ1/detection_loss‚0.5888214383185474 /loss/FRt2/detection_loss‚1.5128417873608047 0loss/FRt2/assignment_loss‚1.1102469081717325 2loss/SRqqt2/assignment_loss‚0.6803049519671318 1loss/SRqqt1/detection_loss‚1.0150366400411686 1loss/SRqqt2/detection_loss‚0.6915632072017285 epoch‚8 trainer/global_step‚7199 % _timestamp‚1.7845047951550252e+09 _runtime‚2949.217846922 0loss/FBt1/assignment_loss‚0.2388497843948926 3loss/FBt2/assignment_loss‚0.0005066364429774458 /loss/FBt1/detection_loss‚0.6279778180146302 /loss/FBt2/detection_loss‚0.5926127650775015 _step‚583 0loss/FRt1/assignment_loss‚2.2886836172405043 &loss/total_loss‚10.000434800472677ÇËýàÕvŸ‚(:ë àÃõÒêö™À! disk.nvme1c1n1.in‚ 10.48828125 disk.md0.out‚ 95.01953125 disk.dm-0.in‚ 110.48046875# disk.dm-0.out‚22355.28759765625 disk.dm-1.in‚0 network.sent‚ 9.471938e+06) disk./.usagePercent‚75.92785960590193( disk.nvme0c0n1.out‚17764.88525390625" disk.dm-1.out‚2933.17041015625 disk.nvme0n1.in‚ 99.9921875' disk.nvme1c1n1.out‚7725.66650390625 disk.nvme1n1.in‚ 10.48828125% disk.nvme1n1.out‚7725.66650390625 disk.md0.in‚0 disk.rbd0.out‚0$ memory_percent‚99.22747611999512' proc.memory.availableMB‚ 126.5703125% disk./.usageGB‚1165.6824645996094 disk.rbd0.in‚0# proc.memory.rssMB‚ 2872.83984375* proc.memory.percent‚17.534422874450684 cpu‚26.458711593056943 proc.cpu.threads‚17 network.recv‚848468 disk.nvme0c0n1.in‚ 99.9921875& disk.nvme0n1.out‚17764.88525390625í¡|› ‚(: àÃõÒÿÄÁÀ gpu.0.gpu‚67.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚111.587' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.19566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0?òT2¡- _wandbruntime‚2968  _runtime‚2968ÚaH¢=¢œ{ 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9994047284126282 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9967120289802551 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03390495479106903 *jet/accuracy_2_of_2‚0.1420200765132904 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07500000298023224 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6864785552024841 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08657599985599518 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.38667550683021545 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7612349390983582 +particle/specificity‚0.9125233888626099 7validation_average_jet_accuracy‚0.33883920311927795 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18864433467388153 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.999609649181366 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.557370662689209 'particle/f_score‚0.6330639719963074 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6215161085128784 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9876849055290222 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6487282514572144 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.603352427482605 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8539987206459045 (particle/accuracy‚0.8478384613990784 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.636733889579773 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6434534192085266 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09088671207427979 jet/accuracy_2_of_5‚1 _runtime‚2968.581998088 /particle/accuracy_4_of_4‚0.9451245069503784 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_5‚1 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.543130099773407 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9967795610427856 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1899605542421341 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 % _timestamp‚1.7845048145184548e+09 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.10624569654464722 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5366446375846863 *jet/accuracy_2_of_4‚0.8028796911239624 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.19644346833229065 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37830430269241333 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.13551294803619385 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_3‚0.9037829041481018 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4175258278846741 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9936224818229675 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15024439990520477 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.038277070969343185 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10750769078731537 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7434922456741333 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7888889312744141 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5430677533149719 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6449056267738342 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4293369650840759 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5469155311584473 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893944263458252 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.10857809334993362 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7231293320655823 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3778029978275299 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0058034565299749374 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8498008251190186 *jet/accuracy_1_of_2‚0.7643933892250061 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11627334356307983 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03926364704966545 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09504343569278717 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6047530174255371 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.014682539738714695 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1237281933426857 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7023965120315552 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9196552038192749 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8799058794975281 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9960682988166809 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9958906769752502 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6084244251251221 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.032130248844623566 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46393781900405884 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.024825027212500572 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8951212763786316 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6224979758262634 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0714285746216774 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2777777910232544 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6644593477249146 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45582258701324463 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.12970487773418427 +jet/accuracy_3_of_4‚0.09502904862165451 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.39772728085517883 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24813611805438995 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.22987651824951172 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6822523474693298 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19356323778629303 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9905549883842468 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.499210000038147 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1209963709115982 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_2_of_4‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9439972639083862 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8421139717102051 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1210181713104248 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7393180131912231 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04145779088139534 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787886738777161 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11562500149011612 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8652881979942322 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5416666865348816 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6420349478721619 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7276236414909363 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 particle/accuracy_1_of_2‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07500000298023224 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07300645858049393 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.13546523451805115 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1209963709115982 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16973130404949188 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.10500887781381607 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +jet/accuracy_3_of_3‚0.06869655847549438 particle/accuracy_2_of_5‚1 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.996148407459259 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8008257746696472 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_5‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7453879117965698 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 +particle/sensitivity‚0.6113206744194031 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.48523762822151184 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 epoch‚8 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6350753903388977 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4973958432674408 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_5_of_5‚0 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.509516179561615 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.750021755695343 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03390495479106903 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9642857313156128 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9965856671333313 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6022799015045166 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013677220791578293 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 trainer/global_step‚7208 particle/accuracy_1_of_1‚1 particle/accuracy_4_of_5‚1 *jet/accuracy_1_of_4‚0.9654839634895325 /particle/accuracy_3_of_3‚0.9994933605194092 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6937483549118042 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7085093855857849 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8618420958518982 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 _step‚584 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2694341242313385 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4749999940395355 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12183971703052521 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8618420958518982 jet/accuracy_5_of_5‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.039692942053079605 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9951990842819214 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9972417950630188 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1269650012254715 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8498008251190186 particle/accuracy_1_of_4‚1 particle/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7334691882133484 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_3‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6434534192085266 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.037673160433769226 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7994227409362793 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.13686847686767578 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6336149573326111 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 (loss/val_total_loss‚9.74185495919079 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14481863379478455 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9903846383094788 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6897892355918884 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010382138192653656 *jet/accuracy_1_of_1‚0.2895454466342926 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8623930811882019 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4844270348548889 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.34876540303230286 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9953703880310059 jet/accuracy_1_of_5‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7614756226539612 particle/accuracy_2_of_2‚1 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9869769811630249 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0714285746216774ü«ÁZ):£¡{ 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.10500887781381607 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 +jet/accuracy_3_of_3‚0.06869655847549438 particle/accuracy_2_of_5‚1 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.996148407459259 & Purity/0FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8008257746696472 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_5‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7453879117965698 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 +particle/sensitivity‚0.6113206744194031 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.48523762822151184 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 epoch‚8 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6350753903388977 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.4973958432674408 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_5_of_5‚0 8"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.509516179561615 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.750021755695343 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03390495479106903 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9642857313156128 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9965856671333313 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6022799015045166 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013677220791578293 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 trainer/global_step‚7208 particle/accuracy_1_of_1‚1 particle/accuracy_4_of_5‚1 *jet/accuracy_1_of_4‚0.9654839634895325 /particle/accuracy_3_of_3‚0.9994933605194092 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6937483549118042 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7085093855857849 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8618420958518982 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.8888888955116272 _step‚584 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2694341242313385 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4749999940395355 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.12183971703052521 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8618420958518982 jet/accuracy_5_of_5‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.039692942053079605 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9951990842819214 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9972417950630188 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1269650012254715 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8498008251190186 particle/accuracy_1_of_4‚1 particle/accuracy_1_of_5‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7334691882133484 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_3‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.7777777910232544 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6434534192085266 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.037673160433769226 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7994227409362793 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.13686847686767578 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6336149573326111 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 (loss/val_total_loss‚9.74185495919079 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14481863379478455 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9903846383094788 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6897892355918884 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.010382138192653656 *jet/accuracy_1_of_1‚0.2895454466342926 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8623930811882019 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4844270348548889 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.34876540303230286 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9953703880310059 jet/accuracy_1_of_5‚1 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7614756226539612 particle/accuracy_2_of_2‚1 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9869769811630249 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0714285746216774 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9994047284126282 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9967120289802551 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03390495479106903 *jet/accuracy_2_of_2‚0.1420200765132904 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.07500000298023224 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6864785552024841 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08657599985599518 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.38667550683021545 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7612349390983582 +particle/specificity‚0.9125233888626099 7validation_average_jet_accuracy‚0.33883920311927795 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18864433467388153 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.999609649181366 8"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.557370662689209 'particle/f_score‚0.6330639719963074 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6215161085128784 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9876849055290222 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6487282514572144 8"Purity/1FRt0SRqqt1FBt/event_purity‚0.603352427482605 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8539987206459045 (particle/accuracy‚0.8478384613990784 8"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.636733889579773 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6434534192085266 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09088671207427979 jet/accuracy_2_of_5‚1 _runtime‚2968.581998088 /particle/accuracy_4_of_4‚0.9451245069503784 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_5‚1 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.543130099773407 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9967795610427856 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.1899605542421341 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 % _timestamp‚1.7845048145184548e+09 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.10624569654464722 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_3‚0.5366446375846863 *jet/accuracy_2_of_4‚0.8028796911239624 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.19644346833229065 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.37830430269241333 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.1818181872367859 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.13551294803619385 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 *jet/accuracy_1_of_3‚0.9037829041481018 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4175258278846741 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 + Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9375 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 /particle/accuracy_3_of_4‚0.9936224818229675 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.15024439990520477 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.038277070969343185 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.10750769078731537 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7434922456741333 ) Purity/*FRt1SRqqt2FBt/FRt_purity‚0.75 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7888889312744141 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5430677533149719 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 & Purity/*FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6449056267738342 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4293369650840759 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5469155311584473 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893944263458252 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.10857809334993362 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7231293320655823 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3778029978275299 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0058034565299749374 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8498008251190186 *jet/accuracy_1_of_2‚0.7643933892250061 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11627334356307983 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03926364704966545 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.09504343569278717 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6047530174255371 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,jet/accuracy_4_of_4‚0.014682539738714695 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1237281933426857 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7023965120315552 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9196552038192749 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8799058794975281 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9960682988166809 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9958906769752502 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6084244251251221 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.032130248844623566 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.46393781900405884 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.024825027212500572 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8951212763786316 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6224979758262634 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0714285746216774 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2777777910232544 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6644593477249146 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45582258701324463 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.12970487773418427 +jet/accuracy_3_of_4‚0.09502904862165451 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.39772728085517883 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24813611805438995 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.22987651824951172 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6822523474693298 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19356323778629303 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9905549883842468 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.499210000038147 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1209963709115982 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 particle/accuracy_2_of_4‚1 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9439972639083862 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8421139717102051 7 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.1210181713104248 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7393180131912231 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04145779088139534 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787886738777161 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11562500149011612 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8652881979942322 jet/accuracy_4_of_5‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 ("Purity/1FRLX~t2SRqqt0FBt/event_purity‚0 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5416666865348816 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6420349478721619 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7276236414909363 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 particle/accuracy_1_of_2‚1 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.07500000298023224 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07300645858049393 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.13546523451805115 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1209963709115982 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.16973130404949188 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0ÈG]‹sh¤‚(:Ý ïÃõÒ“»é¿ disk.dm-0.out‚ 22375.2265625 disk.rbd0.out‚0 disk.nvme0n1.in‚ 99.9921875& disk.nvme0n1.out‚17784.74951171875' proc.memory.availableMB‚ 193.6953125 network.sent‚ 9.521382e+06 disk.nvme1n1.in‚ 10.48828125 disk.md0.in‚0 disk.dm-1.in‚0( disk.nvme0c0n1.out‚17784.74951171875! disk.nvme1c1n1.in‚ 10.48828125) proc.memory.percent‚17.53997802734375( disk./.usagePercent‚75.9320702465053 disk.md0.out‚ 95.23828125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 99.9921875& disk.nvme1c1n1.out‚7790.1884765625$ memory_percent‚98.81777763366699 cpu‚26.470152476348993$ disk.nvme1n1.out‚7790.1884765625 disk.dm-0.in‚ 110.48046875" disk.dm-1.out‚2997.12646484375 proc.memory.rssMB‚2873.75 proc.cpu.threads‚17 network.recv‚852276% disk./.usageGB‚1165.7471084594727¼u\í¥‚(:… ïÃõÒµŒ¿À gpu.0.gpu‚26.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚111.102' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.034 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[ŒgKi¦‚(:Þ þÃõÒ­‚ö¿ disk.rbd0.in‚0( proc.memory.availableMB‚ 193.87109375 cpu‚26.467299020138974 proc.cpu.threads‚17 network.sent‚ 9.579395e+06% disk./.usageGB‚1165.7472381591797 disk.nvme0c0n1.in‚ 99.9921875& disk.nvme0c0n1.out‚17794.638671875$ disk.nvme0n1.out‚17794.638671875 disk.dm-0.in‚ 110.48046875" disk.dm-0.out‚22385.2470703125 disk.rbd0.out‚0 network.recv‚856749 disk.nvme0n1.in‚ 99.9921875! disk.nvme1c1n1.in‚ 10.48828125 disk.md0.in‚0" disk.dm-1.out‚3021.18505859375$ memory_percent‚98.81670475006104) proc.memory.percent‚17.53997802734375 disk.nvme1n1.in‚ 10.48828125% disk.nvme1n1.out‚7814.98291015625 disk.md0.out‚ 95.515625 disk.dm-1.in‚0 proc.memory.rssMB‚2873.75) disk./.usagePercent‚75.93207869462171' disk.nvme1c1n1.out‚7814.98291015625=·k§‚(:… þÃõÒŠ±­À gpu.0.gpu‚49.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.612' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.204 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëlÿ2¨- _wandbruntime‚2985  _runtime‚29858 uÈ©Â trainer/global_step‚7249 % _timestamp‚1.7845048309901347e+09 _runtime‚ 2985.05323441 _step‚585 ' lr-AdamW/pg1‚0.00010482435799552832 ' lr-AdamW/pg2‚0.00010482435799552832ã;s껂 % _timestamp‚1.7845048309901347e+09 _runtime‚ 2985.05323441 _step‚585 ' lr-AdamW/pg1‚0.00010482435799552832 ' lr-AdamW/pg2‚0.00010482435799552832 trainer/global_step‚7249Éhœõµ2«- _wandbruntime‚2985  _runtime‚2985,+Ø'¬ÒO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5105028125696 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5105028125696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31951404 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚279669553013760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73531511 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚279669535974400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128760105 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7845048309910855e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚279669535974400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31951407 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5254175811584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272389410593792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 _step‚586 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5253989106688 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50580371 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚272389198985216 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18628967 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚267135234782208 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚284774669161472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚279669553013760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73541654 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚284774564100096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55218451 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5105116147712 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5089093303562 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18629217 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚267135209878528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚281137236257082 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128749960 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152ïn³·Ý'­×O PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55218451 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73541654 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚281137339444916 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚276048159992880 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚7249 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5105116147712 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55218449 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5089179452036 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5089093303562 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18629217 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚267135209878528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚281137236257082 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128749960 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5105028125696 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5105028125696 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31951404 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚279669553013760 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚73531511 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚279669535974400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128760105 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7845048309910855e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚279669535974400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31951407 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5254175811584 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272389410593792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 _step‚586 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5253989106688 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50580371 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚272389198985216 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18628967 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚267135234782208 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚284774669161472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚279669553013760 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73541654 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚284774564100096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55218451 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5105116147712 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50587132 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128777338 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚267170905035776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚284812713645568 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚279706902341632 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚284812608577024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5089769255006 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5105706235392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194824192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128767179 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 % _timestamp‚1.7845048312579403e+09 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5105794264576 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚281174784276286 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚276085032060640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128777338 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55225772 cODeviceStatsMonitor.on_train_batchý-ÞLŒ_end/inactive_split_bytes.large_pool.allocated‚267170929939456 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18631438 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5254863710720 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55225772 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚272425793650176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5254693790208 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55225770 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚279706902341632 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5105706235392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50587388 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚2985.321333783 _step‚587 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5105794264576 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55225770 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚272425598825984xT×ç&°áM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73551566 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚279706919380992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31955694 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚281174887464232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5105794264576 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚281174784276286 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚276085032060640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128777338 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55225772 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚267170929939456 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18631438 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5254863710720 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55225772 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚272425793650176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5254693790208 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55225770 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚279706902341632 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5105706235392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50587388 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚2985.321333783 _step‚587 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5105794264576 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55225770 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚272425598825984 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚279706919380992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31955697 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚276085015021280 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50587132 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128777338 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚267170905035776 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚284812713645568 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚279706902341632 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚284812608577024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5089769255006 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5105706235392 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194824192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128767179 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 % _timestamp‚1.7845048312579403e+09 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚268919992644608 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚7299 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚281537854336000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚281537871375360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286676820203520 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74028263 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32165907 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5139053889536 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18751368 % _timestamp‚1.7845048502366054e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚286676925264896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74038406 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 _runtime‚3004.302652411 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚277891763380880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚286676820203520 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 _step‚590 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 129612762 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚281537871375360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18751620 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5138965867520 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚283014669120490 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚281537871375360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18751620 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5138965867520 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚283014669120490 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚268919992644608 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚7299 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚281537854336000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚281537871375360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚286676820203520 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74028263 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32165907 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5139053889536 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18751368 % _timestamp‚1.7845048502366054e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚286676925264896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74038406 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104ξýJ2¼- _wandbruntime‚3004  _runtime‚3004ï׳Ùá&½ÛM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55591820 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129640140 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5139732006400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55591822 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55591822 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚286714969748992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚281575220703232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚274245322054144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚268955662898176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32170194 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129629981 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18754041 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚277928618409280 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129640140 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5139643977216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845048505014782e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55591820 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5139732006400 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74048318 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129629981 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50923995 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32170197 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74038161 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚274245535752704 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚268955687801856 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5139643977216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5123684879000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚283052217139694%<çæ&¾àM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚268955662898176 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32170194 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 129629981 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18754041 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚277928618409280 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 129640140 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5139643977216 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845048505014782e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55591820 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5139732006400 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74048318 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 129629981 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50923995 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32170197 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74038161 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚274245535752704 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚268955687801856 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5139643977216 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5123684879000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚283052217139694 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚55591820 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 129640140 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5139732006400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55591822 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55591822 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚286714969748992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚281575220703232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚274245322054144 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330ÏÊŠ‡2¿- _wandbruntime‚3004  _runtime‚3004»i mÀ€ 1loss/SRqqt1/detection_loss‚1.1622409770711206 trainer/global_step‚7299 _step‚592 .loss/FBt2/detection_loss‚0.534272058586043 epoch‚9 2loss/SRqqt1/assignment_loss‚0.7716164979186507 % _timestamp‚1.7845048505024793e+09 _runtime‚ 3004.56544852 1loss/SRqqt2/detection_loss‚0.7755279436312604 &loss/total_loss‚10.324203702878695 /loss/FRt1/assignment_loss‚2.393836202715883 2loss/SRqqt2/assignment_loss‚0.6161517174352749 1loss/FBt1/assignment_loss‚0.05558021575443274 2loss/FBt2/assignment_loss‚0.007391072025746051 /loss/FRt1/detection_loss‚0.5699026556491638 /loss/FRt2/detection_loss‚1.5246174618199901 /loss/FBt1/detection_loss‚0.9664094706788872 0loss/FRt2/assignment_loss‚0.9466574295922434ÂÍTÅ Á… &loss/total_loss‚10.324203702878695 /loss/FRt1/assignment_loss‚2.393836202715883 2loss/SRqqt2/assignment_loss‚0.6161517174352749 1loss/FBt1/assignment_loss‚0.05558021575443274 2loss/FBt2/assignment_loss‚0.007391072025746051 /loss/FRt1/detection_loss‚0.5699026556491638 /loss/FRt2/detection_loss‚1.5246174618199901 /loss/FBt1/detection_loss‚0.9664094706788872 0loss/FRt2/assignment_loss‚0.9466574295922434 1loss/SRqqt1/detection_loss‚1.1622409770711206 trainer/global_step‚7299 _step‚592 .loss/FBt2/detection_loss‚0.534272058586043 epoch‚9 2loss/SRqqt1/assignment_loss‚0.7716164979186507 % _timestamp‚1.7845048505024793e+09 _runtime‚ 3004.56544852 1loss/SRqqt2/detection_loss‚0.7755279436312604РÖe‚(:Ú œÄõÒø¯Ò¿& disk.nvme0n1.out‚17809.74853515625 disk.md0.out‚ 96.546875 network.sent‚ 9.645438e+06 network.recv‚864015 disk.dm-0.in‚ 110.48046875 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme1n1.in‚ 10.48828125 disk.md0.in‚0 proc.memory.rssMB‚2873.75# disk.dm-0.out‚22399.72802734375! disk.dm-1.out‚3061.4384765625 disk.nvme0c0n1.in‚ 99.9921875! disk.nvme1c1n1.in‚ 10.48828125% disk.nvme1c1n1.out‚7856.794921875$ memory_percent‚98.81718158721924( proc.memory.availableMB‚ 193.79296875) proc.memory.percent‚17.53997802734375' disk./.usagePercent‚75.932084906472 disk.nvme0n1.in‚ 99.9921875# disk.nvme1n1.out‚7856.794921875 cpu‚26.4616030345028 proc.cpu.threads‚17% disk./.usageGB‚1165.7473335266113 disk.rbd0.out‚0( disk.nvme0c0n1.out‚17809.74853515625>³õš›Ã‚(: œÄõÒƒ­«À gpu.0.gpu‚63.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.601' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.20033333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žÙö?2Ä- _wandbruntime‚3023  _runtime‚3023N·åÉÅà _step‚593 ' lr-AdamW/pg1‚0.00010460133959411205 ' lr-AdamW/pg2‚0.00010460133959411205 trainer/global_step‚7349 % _timestamp‚1.7845048698037853e+09 _runtime‚3023.866920494V»ÎÆÈ % _timestamp‚1.7845048698037853e+09 _runtime‚3023.866920494 _step‚593 ' lr-AdamW/pg1‚0.00010460133959411205 ' lr-AdamW/pg2‚0.00010460133959411205 trainer/global_step‚7349Ñê®ou2Ç- _wandbruntime‚3023  _runtime‚3023_}´©ÈÒO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55950549 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚270704750507008 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚284892205171732 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276028684069376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51255552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚288579181368320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚283406172697600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74535158 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚284892101983898 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚279735366768880 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚3023.867747475 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚283406172697600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51255307 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55950549 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74525015 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18874903 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5156752254378 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5324138879488 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 130475564 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚283406189736960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55950551 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚283406189736960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5172903609344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276028889386496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74525015 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batchZöåž/_start/inactive_split_bytes.all.peak‚ 771838464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚279735349729520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5172991631360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130485709 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5156838402852 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚288579181368320 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚288579076306944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚270704725603328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚288579076306944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5172903609344 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 trainer/global_step‚7349 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 _step‚594 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32380404 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18875145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360;ƒßÝ'É×O LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74525015 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚279735349729520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5172991631360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130485709 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5156838402852 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚288579181368320 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚288579076306944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚270704725603328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚288579076306944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5172903609344 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 trainer/global_step‚7349 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 _step‚594 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32380404 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18875145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55950549 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚270704750507008 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚284892205171732 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚276028684069376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51255552 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚288579181368320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚283406172697600 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74535158 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚284892101983898 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚279735366768880 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚3023.867747475 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚283406172697600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51255307 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55950549 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74525015 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18874903 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5156752254378 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5324138879488 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 130475564 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚283406189736960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55950551 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚283406189736960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5172903609344 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚276028889386496 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448ÒXit2Ê- _wandbruntime‚3024  _runtime‚3024Ñ°]~á&ËÛM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚284929650003102 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5173669748224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32384694 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚288617225852416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚283443539064832 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚284929753191048 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5157514354408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130502942 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74545070 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5173581719040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 $ _timestamp‚1.784504870070002e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74534913 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚283443556104192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74534913 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚276065282064384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55957872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5157428205822 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 _runtime‚3024.133280023 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18877722 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚288617120783872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚270740420760576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130502942 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74545070 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5173581719040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18877460 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚288617225852416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚279772221797280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55957872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32384697 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5173669748224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51262419 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5324651799552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5324836400128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51262154 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚283443556104192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚276065072560128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚279772238836640 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 trainer/global_step‚7349 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482:ôt¾"ÌàM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚284929650003102 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5173669748224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32384694 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚288617225852416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚283443539064832 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚284929753191048 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5157514354408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130502942 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74545070 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5173581719040 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 $ _timestamp‚1.784504870070002e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74534913 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚283443556104192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74534913 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚276065282064384 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55957872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5157428205822 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 _runtime‚3024.133280023 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18877722 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚288617120783872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚270740420760576 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 130502942 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74545070 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5173581719040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18877460 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚288617225852416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚279772221797280 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚55957872 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32384697 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5173669748224 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51262419 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5324651799552 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5324836400128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51262154 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚283443556104192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚276065072560128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚279772238836640 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 trainer/global_step‚7349 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208Ó£¨in2Í- _wandbruntime‚3024  _runtime‚3024˜§}8΀ 0loss/FRt1/assignment_loss‚2.3401196427673465 2loss/FBt2/assignment_loss‚0.003456573577891078 0loss/SRqqt1/detection_loss‚1.245772656566632 1loss/SRqqt2/detection_loss‚0.6662081031178789 /loss/FBt2/detection_loss‚0.6138303143810481 trainer/global_step‚7349 $ _timestamp‚1.784504870070989e+09 0loss/FRt2/assignment_loss‚0.9985323763086729 2loss/SRqqt1/assignment_loss‚0.7554722870956725 /loss/FRt2/detection_loss‚1.5395631462124437 &loss/total_loss‚10.132928380380054 2loss/SRqqt2/assignment_loss‚0.6665563911246816 1loss/FBt1/assignment_loss‚0.01334691872180945 .loss/FBt1/detection_loss‚0.732866213346521 _runtime‚3024.133811804 _step‚596 /loss/FRt1/detection_loss‚0.5572037571594551 epoch‚9>ýP’ Ï… epoch‚9 0loss/FRt1/assignment_loss‚2.3401196427673465 2loss/FBt2/assignment_loss‚0.003456573577891078 0loss/SRqqt1/detection_loss‚1.245772656566632 1loss/SRqqt2/detection_loss‚0.6662081031178789 /loss/FBt2/detection_loss‚0.6138303143810481 trainer/global_step‚7349 $ _timestamp‚1.784504870070989e+09 0loss/FRt2/assignment_loss‚0.9985323763086729 2loss/SRqqt1/assignment_loss‚0.7554722870956725 /loss/FRt2/detection_loss‚1.5395631462124437 &loss/total_loss‚10.132928380380054 2loss/SRqqt2/assignment_loss‚0.6665563911246816 1loss/FBt1/assignment_loss‚0.01334691872180945 .loss/FBt1/detection_loss‚0.732866213346521 _runtime‚3024.133811804 _step‚596 /loss/FRt1/detection_loss‚0.5572037571594551Ô› @kЂ(:à «ÄõÒºÓò¿ disk.md0.in‚0$ memory_percent‚98.81758689880371 cpu‚26.45883102576185 disk.rbd0.in‚0( disk.nvme0c0n1.out‚17839.06494140625 disk.md0.out‚97.09375 proc.cpu.threads‚17 network.sent‚ 9.721631e+06) disk./.usagePercent‚75.93208242173188 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 99.9921875 disk.nvme0n1.in‚ 99.9921875! disk.nvme1c1n1.in‚ 10.48828125' disk.nvme1c1n1.out‚7857.63037109375 disk.nvme1n1.in‚ 10.48828125% disk./.usageGB‚1165.7472953796387& disk.nvme0n1.out‚17839.06494140625% disk.nvme1n1.out‚7857.63037109375 disk.dm-0.in‚ 110.48046875# disk.dm-0.out‚22428.54248046875' proc.memory.availableMB‚ 193.7265625 proc.memory.rssMB‚2873.75) proc.memory.percent‚17.53997802734375 network.recv‚868421" disk.dm-1.out‚3061.60791015625 disk.rbd0.out‚0Åoq›Ñ‚(: «ÄõÒ ®›À gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚111.709' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.236333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0KI’2Ò- _wandbruntime‚3043  _runtime‚3043ü@) ÉÓà _runtime‚3043.437832267 _step‚597 ' lr-AdamW/pg1‚0.00010448866334942555 ' lr-AdamW/pg2‚0.00010448866334942555 trainer/global_step‚7399 % _timestamp‚1.7845048893746903e+090»DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚290481437471744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51593562 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚281578970156880 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚272489508369408 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚290481437471744 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚286769638035140 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75031910 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚290481332410368 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚286769534847306 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 trainer/global_step‚7399 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56316599 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5359129370624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5190581729786 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56316601 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚272489483465728 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5206841351168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75031910 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _runtime‚3043.438671868 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚290481332410368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚285274491059200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56316599 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5190667878260 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75021767 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32594907 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚285274508098560 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5206841351168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75031910 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _runtime‚3043.438671868 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚290481332410368 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚285274491059200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚56316599 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5190667878260 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75021767 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32594907 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚285274508098560 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚290481437471744 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51593562 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚281578970156880 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚272489508369408 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚290481437471744 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚286769638035140 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75031910 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚290481332410368 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚286769534847306 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 trainer/global_step‚7399 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56316599 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5359129370624 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5190581729786 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56316601 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚272489483465728 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5191257681230 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚599 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚277885024949760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚277884823834112 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75041822 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32599197 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 trainer/global_step‚7399 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚285311874465792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131365744 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _runtime‚3043.705153658 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131365744 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56323922 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56323920 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚285311857426432 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚290519376887296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131355585 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚286807186054456 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75031665 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56323922 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51600098 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚286807082866510 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75041822 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚290519481955840 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚285311874465792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batvݵ—ch_end/inactive_split.large_pool.freed‚32599194 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5207607490048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚281615825185280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚290519481955840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75031665 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5359821423104 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5207607490048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚272525178622976 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚281615842224640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚3304}ýÑç&ÚáM FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚599 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚277885024949760 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚277884823834112 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚75041822 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32599197 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 trainer/global_step‚7399 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚285311874465792 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131365744 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _runtime‚3043.705153658 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131365744 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56323922 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚56323920 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚285311857426432 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚290519376887296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131355585 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚286807186054456 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75031665 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56323922 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51600098 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚286807082866510 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75041822 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚290519481955840 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚285311874465792 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32599194 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5207607490048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚281615825185280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚290519481955840 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75031665 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5359821423104 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5207607490048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚272525178622976 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚281615842224640 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚272525203526656 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5207519460864 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5359645211136 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5191257681230 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ES¿ÎäÈ _runtime‚3063.008996055 _step‚601 ' lr-AdamW/pg1‚0.00010437521147758779 ' lr-AdamW/pg2‚0.00010437521147758779 trainer/global_step‚7449 % _timestamp‚1.7845049089458637e+09Ù‰‰5,2å- _wandbruntime‚3063  _runtime‚3063Pºg×'æÑO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56682651 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚602 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56682651 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19122744 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 $ _timestamp‚1.784504908946679e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32809404 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚274274266231808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32809407 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75518519 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚292383693575168 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5394139626496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚288647070898548 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚274274241328128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5240867115008 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 132201168 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚287142809420800 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199025664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292383588513792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚283422573544880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75528662 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚292383588513792 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75518519 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5240779092992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5240779092992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51931912 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5224497353668 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5240867115008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚3063.011169549 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚283422556505520 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292383588513792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚283422573544880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75528662 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚236 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚292383588513792 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75518519 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5240779092992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5240779092992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51931912 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5224497353668 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5240867115008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚3063.011169549 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚283422556505520 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚239 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚56682651 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚602 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56682651 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19122744 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 $ _timesÞdHätamp‚1.784504908946679e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32809404 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚274274266231808 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32809407 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚75518519 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚292383693575168 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5394139626496 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚288647070898548 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚274274241328128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5240867115008 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 132201168 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚287142809420800 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199025664 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474Ú }yR2è- _wandbruntime‚3063  _runtime‚3063 Æš‚â&éÜM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚288684515729918 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚287180192827392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5241545231872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5225173305224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56689972 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132228546 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5394663097856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 trainer/global_step‚7449 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75538574 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚283459428573280 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚292421738059264 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5241457202688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7845049092008617e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75528417 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚279704802796032 _runtime‚3063.264151453 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚288684618917864 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32813694 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283459445612640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚287180192827392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56689970 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51938573 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32813697 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75528417 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚292421738059264 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5241545231872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51938826 _step‚603 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚287180175788032 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 132218387 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚292421632990720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚274309936485376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚274309961389056 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5225087156638 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚287180175788032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374'OYç&êáM EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚279704802796032 _runtime‚3063.264151453 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚288684618917864 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32813694 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚283459445612640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚287180192827392 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56689970 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51938573 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32813697 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75528417 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚292421738059264 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5241545231872 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51938826 _step‚603 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚287180175788032 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 132218387 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚292421632990720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚274309936485376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚274309961389056 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5225087156638 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚287180175788032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚288684515729918 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚287180192827392 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5241545231872 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5225173305224 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚56689972 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 132228546 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5394663097856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 trainer/global_step‚7449 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75538574 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚283459428573280 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚292421738059264 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5241457202688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7845049092008617e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚75528417 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808Û à2ë- _wandbruntime‚3063  _runtime‚3063o½Þ˜ì .loss/FBt2/detection_loss‚0.520866010702836 _runtime‚3063.264595213 _step‚604 /loss/FRt1/assignment_loss‚2.509038349274199 2loss/SRqqt2/assignment_loss‚0.7456066264425185 /loss/FRt2/detection_loss‚1.5180874678707268 1loss/SRqqt1/detection_loss‚1.0889936834177438 epoch‚9 trainer/global_step‚7449 2loss/SRqqt1/assignment_loss‚0.8228157882063472 2loss/FBt1/assignment_loss‚0.002147669717868192 /loss/FRt1/detection_loss‚0.5472625970339473 1loss/SRqqt2/detection_loss‚0.6974664234832568 &loss/total_loss‚10.422619672069384 % _timestamp‚1.7845049092017636e+09 0loss/FRt2/assignment_loss‚1.0291486188532815 1loss/FBt2/assignment_loss‚0.09014698669396767 /loss/FBt1/detection_loss‚0.8510394503726907W;‚ í† 1loss/SRqqt1/detection_loss‚1.0889936834177438 epoch‚9 trainer/global_step‚7449 2loss/SRqqt1/assignment_loss‚0.8228157882063472 2loss/FBt1/assignment_loss‚0.002147669717868192 /loss/FRt1/detection_loss‚0.5472625970339473 1loss/SRqqt2/detection_loss‚0.6974664234832568 &loss/total_loss‚10.422619672069384 % _timestamp‚1.7845049092017636e+09 0loss/FRt2/assignment_loss‚1.0291486188532815 1loss/FBt2/assignment_loss‚0.09014698669396767 /loss/FBt1/detection_loss‚0.8510394503726907 .loss/FBt2/detection_loss‚0.520866010702836 _runtime‚3063.264595213 _step‚604 /loss/FRt1/assignment_loss‚2.509038349274199 2loss/SRqqt2/assignment_loss‚0.7456066264425185 /loss/FRt2/detection_loss‚1.5180874678707268ܲƒÜ“lî‚(:á ØÄõÒåžÂÁ( disk.nvme0c0n1.out‚17852.99267578125 disk.nvme0n1.in‚ 99.9921875! disk.nvme1c1n1.in‚ 10.48828125 disk.rbd0.in‚0) proc.memory.percent‚17.53997802734375 proc.cpu.threads‚17 network.recv‚880385) disk./.usagePercent‚75.94850779627961' disk.nvme1c1n1.out‚7859.71044921875# disk.dm-0.out‚22442.42041015625" memory_percent‚98.818039894104( proc.memory.availableMB‚ 193.65234375 network.sent‚ 9.837232e+06 disk.nvme1n1.in‚ 10.48828125% disk.nvme1n1.out‚7859.71044921875 disk.dm-0.in‚ 110.48046875 disk.dm-1.in‚0" disk.dm-1.out‚3061.84130859375 proc.memory.rssMB‚2873.75 cpu‚26.45300833736835 disk.nvme0c0n1.in‚ 99.9921875& disk.nvme0n1.out‚17852.99267578125 disk.md0.in‚0 disk.md0.out‚ 97.9609375 disk.rbd0.out‚0% disk./.usageGB‚1165.9994659423828ªôÐd›ï‚(: ØÄõÒœÑú¿ gpu.0.gpu‚66.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚193.234' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚64.41133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†æ 42ð- _wandbruntime‚3082  _runtime‚3082¿X¾,Èñ ' lr-AdamW/pg1‚0.00010426098579576078 ' lr-AdamW/pg2‚0.00010426098579576078 trainer/global_step‚7499 $ _timestamp‚1.784504928117846e+09 _runtime‚3082.181013833 _step‚605I¬)7ÍòÇ ' lr-AdamW/pg2‚0.00010426098579576078 trainer/global_step‚7499 $ _timestamp‚1.784504928117846e+09 _runtime‚3082.181013833 _step‚605 ' lr-AdamW/pg1‚0.00010426098579576078Ýò‘”~2ó- _wandbruntime‚3082  _runtime‚3082ñ%•ÍzôÑO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57048699 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5274716834816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57048701 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚281488168573952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5274804856832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚285266159893520 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133063970 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚294285844617216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5258326829076 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚281487967451136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚294285949678592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33023904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚276059024094208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5428968260608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76015271 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚276058999190528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚289011144821760 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚294285844617216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚289011127782400 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚294285949678592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚289011127782400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚290524503761956 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 $ _timestamp‚1.784504928118931e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 _step‚606 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76015271 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚7499'$f_Ü'õÖO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57048699 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚290524400574122 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5429144479744 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76025414 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5274716834816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201122816 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚285266176932880 _runtime‚3082.182597256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚289011144821760 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133074115 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76025414 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚276059024094208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5428968260608 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76015271 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚276058999190528 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚289011144821760 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚294285844617216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚289011127782400 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚294285949678592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚289011127782400 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚290524503761956 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 $ _timestamp‚1.784504928118931e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 _step‚606 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76015271 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚7499 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57048699 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5274716834816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57048701 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚281488168573952 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5274804856832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚285266159893520 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133063970 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚294285844617216 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5258326829076 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚281487967451136 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚294285949678592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33023904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19247531 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133091348 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76025169 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚276094719251456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚276094694347776 _runtime‚3082.762025602 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33028197 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _step‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚289048494149632 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5275482973696 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5275394944512 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚290562051781272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 trainer/global_step‚7499 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚289048494149632 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19247786 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5275394944512 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76025169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52275725 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚294323889094144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚285303049000640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33028194 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚289048511188992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚294323994162688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133081189 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57056022 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7845049286986618e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚294323994162688 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57056020 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290561948593326 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5258916632046 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5275482973696 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5429663166976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52275725 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚294323889094144 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚285303049000640 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33028194 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚289048511188992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚294323994162688 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133081189 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57056022 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7845049286986618e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚294323994162688 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57056020 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290561948593326 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5258916632046 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5275482973696 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5429663166976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19247531 BDeviceStatsMonitor.on_train_batch_end/allocated_ÍkõsW bytes.all.peak‚ 6159996928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133091348 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76025169 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚276094719251456 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚276094694347776 _runtime‚3082.762025602 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33028197 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _step‚607 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚289048494149632 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5275482973696 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5275394944512 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚290562051781272 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 trainer/global_step‚7499 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚289048494149632 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19247786 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5275394944512 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76025169 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184ß‹¤²2ù- _wandbruntime‚3082  _runtime‚3082Î€È ' lr-AdamW/pg1‚0.00010414598813350066 ' lr-AdamW/pg2‚0.00010414598813350066 trainer/global_step‚7549 % _timestamp‚1.7845049482435198e+09 _runtime‚3102.308123746 _step‚609á‰h›{2- _wandbruntime‚3102  _runtime‚3102+®ÜØ'‚ÒO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚283307977246208 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚287109780320880 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33238404 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52606615 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5292156304484 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57414749 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚290879463183360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57414749 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5308742598656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5308742598656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5308654576640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚287109763281520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133926772 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19367966 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296188100720640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52606370 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33238407 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19368208 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133936917 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76522166 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57414751 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76512023 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚292401936625364 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5464195289600 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚292401833437530 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚290879446144000 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚283307774026240 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚277843781956608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 trainer/global_step‚7549 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚290879463183360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚290879446144000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5292070156010 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76522166 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚296188100720640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57414751 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 % _timestamp‚1.7845049482459497e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76512023ÙFÉÝ'ƒ×O VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚296188205782016 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5308654576640 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33238404 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52606615 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5292156304484 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57414749 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚290879463183360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57414749 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5308742598656 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5308742598656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5308654576640 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚287109763281520 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 133926772 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19367966 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296188100720640 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52606370 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33238407 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19368208 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133936917 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚76522166 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57414751 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76512023 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚292401936625364 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5464195289600 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚292401833437530 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚290879446144000 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚283307774026240 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚277843781956608 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 trainer/global_step‚7549 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚290879463183360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚290879446144000 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5292070156010 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76522166 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚296188100720640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57414751 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 % _timestamp‚1.7845049482459497e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76512023 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚283307977246208 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚287109780320880 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133954150 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚287146652388640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57422072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5464898204672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5292746107454 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 _step‚611 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76532078 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚277879452210176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7845049485015008e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52613053 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚283344165814272 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5292832256040 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚296226250266112 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚296226145197568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚277879477113856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚292439381456734 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57422072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57422070 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76521921 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5309420715520 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133943991 L=DevŠnJ¦ÑiceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚290916829550592 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚283344375318528 trainer/global_step‚7549 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76521921 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚296226250266112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5309332686336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19370359 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3102.564815583 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33242694 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5464713604096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚277879452210176 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 % _timestamp‚1.7845049485015008e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52613053 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚283344165814272 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5292832256040 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚296226250266112 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚296226145197568 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚277879477113856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚292439381456734 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57422072 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57422070 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76521921 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚244 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5309420715520 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133943991 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚290916829550592 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚283344375318528 trainer/global_step‚7549 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚76521921 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚296226250266112 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5309332686336 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19370359 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3102.564815583 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33242694 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5464713604096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133954150 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚287146652388640 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57422072 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5464898204672 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5292746107454ãa†©î2‡- _wandbruntime‚3102  _runtime‚3102í0Ô¥ˆÿ 0loss/FBt1/assignment_loss‚0.1211505350232843 1loss/SRqqt2/detection_loss‚0.6778846809094246 &loss/total_loss‚10.410070950982618 epoch‚9 trainer/global_step‚7549 0loss/FRt2/assignment_loss‚1.0333228343315708 2loss/SRqqt1/assignment_loss‚0.7970694935667031 /loss/FRt2/detection_loss‚1.4621422712527614 0loss/SRqqt1/detection_loss‚1.024823589483276 /loss/FBt2/detection_loss‚0.5525115866315197 _runtime‚3102.566027815 _step‚612 /loss/FRt1/assignment_loss‚2.377543653134538 1loss/FBt2/assignment_loss‚0.12907161730620278 /loss/FRt1/detection_loss‚0.6288884389148642 /loss/FBt1/detection_loss‚1.0190251501994003 % _timestamp‚1.7845049485031433e+09 2loss/SRqqt2/assignment_loss‚0.58663710022907192æ⧠‰„ 0loss/FBt1/assignment_loss‚0.1211505350232843 1loss/SRqqt2/detection_loss‚0.6778846809094246 &loss/total_loss‚10.410070950982618 epoch‚9 trainer/global_step‚7549 0loss/FRt2/assignment_loss‚1.0333228343315708 2loss/SRqqt1/assignment_loss‚0.7970694935667031 /loss/FRt2/detection_loss‚1.4621422712527614 0loss/SRqqt1/detection_loss‚1.024823589483276 /loss/FBt2/detection_loss‚0.5525115866315197 _runtime‚3102.566027815 _step‚612 /loss/FRt1/assignment_loss‚2.377543653134538 1loss/FBt2/assignment_loss‚0.12907161730620278 /loss/FRt1/detection_loss‚0.6288884389148642 /loss/FBt1/detection_loss‚1.0190251501994003 % _timestamp‚1.7845049485031433e+09 2loss/SRqqt2/assignment_loss‚0.5866371002290719ä>-«Þ›Š‚(: öÄõÒë¦Ó¿ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚149.344' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚49.781333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸ž;i‹‚(:Þ öÄõÒ¥øÿ¿ proc.cpu.threads‚17 network.sent‚ 9.961917e+06 network.recv‚888978 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 99.9921875% disk.nvme1c1n1.out‚7861.701171875 disk.nvme1n1.in‚ 10.48828125# disk.nvme1n1.out‚7861.701171875) disk./.usagePercent‚75.93310837092646% disk./.usageGB‚1165.7630462646484 disk.rbd0.out‚0& disk.nvme0n1.out‚17884.75537109375! disk.nvme1c1n1.in‚ 10.48828125 disk.dm-0.in‚ 110.48046875" disk.dm-0.out‚22473.7529296875( proc.memory.availableMB‚ 128.15234375" disk.dm-1.out‚3062.03466796875 disk.nvme0c0n1.in‚ 99.9921875 disk.md0.in‚0 cpu‚26.45949852300852( disk.nvme0c0n1.out‚17884.75537109375 disk.md0.out‚ 99.01171875# memory_percent‚99.2178201675415 proc.memory.rssMB‚2873.75) proc.memory.percent‚17.53997802734375ùë]áiŒ‚(:Þ …ÅõÒæƒß¿ proc.cpu.threads‚17$ disk./.usageGB‚1165.763126373291" disk.dm-0.out‚22474.9716796875% disk.nvme0n1.out‚17886.3408203125! disk.nvme1c1n1.in‚ 10.48828125$ disk.nvme1n1.out‚7862.3330078125 disk.md0.out‚ 99.43359375 network.sent‚ 1.0026224e+07( disk./.usagePercent‚75.9331135888807 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 99.9921875' disk.nvme0c0n1.out‚17886.3408203125 disk.nvme0n1.in‚ 99.9921875 proc.memory.rssMB‚2873.75 disk.dm-0.in‚ 110.48046875 disk.dm-1.in‚0! disk.dm-1.out‚3062.1357421875 disk.rbd0.in‚0& disk.nvme1c1n1.out‚7862.3330078125 disk.md0.in‚0$ memory_percent‚99.21643733978271) proc.memory.percent‚17.53997802734375 network.recv‚893387 disk.nvme1n1.in‚ 10.48828125( proc.memory.availableMB‚ 128.37890625 cpu‚26.461850663508162ÙZÂ(:… …ÅõÒÙ«ˆÀ gpu.0.gpu‚79.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚134.955' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚44.985 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ã»D 2Ž- _wandbruntime‚3121  _runtime‚3121Á6©Éà ' lr-AdamW/pg2‚0.00010403022033272839 trainer/global_step‚7599 % _timestamp‚1.7845049677456124e+09 _runtime‚3121.808748417 _step‚613 ' lr-AdamW/pg1‚0.00010403022033272839ÅdÝÎÈ _step‚613 ' lr-AdamW/pg1‚0.00010403022033272839 ' lr-AdamW/pg2‚0.00010403022033272839 trainer/global_step‚7599 % _timestamp‚1.7845049677456124e+09 _runtime‚3121.808748417å{Ëv2‘- _wandbruntime‚3121  _runtime‚3121ð&>”Ø'’ÒO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33452904 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7845049677465315e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 134789574 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52944415 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57780801 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚292747764505600 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19491508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5342680340480 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚298090356824064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚279628539819008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚288953366669520 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5325985779892 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚3121.809549718 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134799719 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57780799 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5499008959488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚285127727094784 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33452907 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5342680340480 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77018918 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77008775 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 trainer/global_step‚7599 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚298090461885440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚298090356824064 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚294279266300938 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚292747781544960 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5342592318464 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _step‚614 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚292747781544960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚298090461885440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5342592318464 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitë˜)or.on_train_batch_start/active.small_pool.current‚10143 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 % _timestamp‚1.7845049677465315e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 134789574 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52944415 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57780801 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚292747764505600 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19491508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5342680340480 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚298090356824064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚279628539819008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚288953366669520 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5325985779892 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚3121.809549718 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 134799719 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57780799 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5499008959488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚285127727094784 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33452907 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5342680340480 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77018918 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚77008775 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 trainer/global_step‚7599 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚298090461885440 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚298090356824064 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚294279266300938 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚292747781544960 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5342592318464 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _step‚614 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482æ—N}ê2”- _wandbruntime‚3122  _runtime‚31223D)Já&•ÛM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5343358457344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19493704 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚288990238737280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5326661731448 _runtime‚3122.063626043 _step‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚298128506369536 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77028830 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134806793 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134816952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52950898 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 trainer/global_step‚7599 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77018673 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚285163922384384 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52951138 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57788120 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5343270428160 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚298128506369536 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚294316917508088 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚288990255776640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚292785130872832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5499712311808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 134816952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚298128401300992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚292785130872832 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77018673 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33457197 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚294316814320142 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57788120 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚279664210072576 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5343358457344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57788122 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57788122 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚292785147912192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5326575582862 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 134806793 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19493941 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5499886426624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174114816 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ŠJæ&–àM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚288990238737280 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5326661731448 _runtime‚3122.063626043 _step‚615 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚298128506369536 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77028830 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 134806793 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 134816952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52950898 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 trainer/global_step‚7599 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77018673 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚285163922384384 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52951138 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57788120 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5343270428160 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚298128506369536 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚294316917508088 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚288990255776640 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚292785130872832 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5499712311808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 134816952 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚298128401300992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚292785130872832 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77018673 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33457197 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚294316814320142 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57788120 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚279664210072576 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5343358457344 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57788122 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57788122 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚292785147912192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5326575582862 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 134806793 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19493941 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5499886426624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174114816 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5343358457344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19493704çã9ä 2—- _wandbruntime‚3122  _runtime‚3122§%þܘ€ 0loss/SRqqt1/detection_loss‚1.105999016034223 1loss/SRqqt2/detection_loss‚0.7812197598643122 /loss/FBt1/detection_loss‚1.1035747194991392 epoch‚9 trainer/global_step‚7599 0loss/FRt2/assignment_loss‚1.1667506566335415 2loss/SRqqt2/assignment_loss‚0.7587314968702732 3loss/FBt2/assignment_loss‚0.0038493211419486644 /loss/FBt2/detection_loss‚0.4439608373366235 _runtime‚3122.064116713 /loss/FRt1/assignment_loss‚2.365601535821926 &loss/total_loss‚11.022843556700295 0loss/FBt1/assignment_loss‚0.2435599748029798 /loss/FRt1/detection_loss‚0.6245046663497024 /loss/FRt2/detection_loss‚1.6712288574881626 % _timestamp‚1.7845049680013072e+09 _step‚616 1loss/SRqqt1/assignment_loss‚0.753862714857463e¯ ™… 0loss/FBt1/assignment_loss‚0.2435599748029798 /loss/FRt1/detection_loss‚0.6245046663497024 /loss/FRt2/detection_loss‚1.6712288574881626 % _timestamp‚1.7845049680013072e+09 _step‚616 1loss/SRqqt1/assignment_loss‚0.753862714857463 0loss/SRqqt1/detection_loss‚1.105999016034223 1loss/SRqqt2/detection_loss‚0.7812197598643122 /loss/FBt1/detection_loss‚1.1035747194991392 epoch‚9 trainer/global_step‚7599 0loss/FRt2/assignment_loss‚1.1667506566335415 2loss/SRqqt2/assignment_loss‚0.7587314968702732 3loss/FBt2/assignment_loss‚0.0038493211419486644 /loss/FBt2/detection_loss‚0.4439608373366235 _runtime‚3122.064116713 /loss/FRt1/assignment_loss‚2.365601535821926 &loss/total_loss‚11.022843556700295è$€/lkš‚(:à ”ÅõÒ¨‹À network.sent‚ 1.0041989e+07% disk./.usageGB‚1165.7631034851074! disk.dm-1.out‚3062.2568359375' disk.nvme0c0n1.out‚17902.1025390625 disk.nvme0n1.in‚ 99.9921875! disk.nvme1c1n1.in‚ 10.48828125' proc.memory.availableMB‚ 128.3515625 proc.memory.rssMB‚2873.75 network.recv‚896827 disk.dm-0.in‚ 110.48046875 disk.rbd0.out‚0 disk.md0.out‚ 100.18359375) proc.memory.percent‚17.53997802734375) disk./.usagePercent‚75.93311209803663 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 99.9921875% disk.nvme0n1.out‚17902.1025390625 disk.nvme1n1.in‚ 10.48828125$ disk.nvme1n1.out‚7864.8935546875 disk.md0.in‚0" disk.dm-0.out‚22491.6142578125 disk.dm-1.in‚0& disk.nvme1c1n1.out‚7864.8935546875$ memory_percent‚99.21660423278809 cpu‚26.465916335814175 proc.cpu.threads‚17¹Ã­››‚(: ”ÅõÒ³ô°À gpu.0.gpu‚72.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚189.91' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚63.303333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹]Ý2œ- _wandbruntime‚3141  _runtime‚3141ßP}–ÇÁ trainer/global_step‚7649 % _timestamp‚1.7845049869975412e+09 _runtime‚3141.060740556 _step‚617 & lr-AdamW/pg1‚0.0001039136842477003 & lr-AdamW/pg2‚0.0001039136842477003ßÝûÌžÆ _step‚617 & lr-AdamW/pg1‚0.0001039136842477003 & lr-AdamW/pg2‚0.0001039136842477003 trainer/global_step‚7649 % _timestamp‚1.7845049869975412e+09 _runtime‚3141.060740556éÍêÄ—2Ÿ- _wandbruntime‚3141  _runtime‚3141F ˆqV  ÑO UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚296156699164346 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 trainer/global_step‚7649 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53282256 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58146851 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135662521 _step‚618 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚227 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚286947314839040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚286947509670400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169927680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5534042061312 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚290796987096880 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58146849 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33667407 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚296156802352180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocate[,¶Nd_bytes.small_pool.freed‚ 5376530060288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚299992612927488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5376618082304 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚294616099906560 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 $ _timestamp‚1.784504986998257e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77505527 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5359729106826 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58146849 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77515670 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚294616082867200 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77515670 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19615076 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5359815255300 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚294616082867200 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚281413272777728 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19614852 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 _runtime‚3141.061255807 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5376530060288 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0úû¸Ü'¡ÖO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚299992612927488 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5376618082304 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚294616099906560 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 $ _timestamp‚1.784504986998257e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚77505527 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5359729106826 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58146849 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77515670 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚294616082867200 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77515670 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19615076 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5359815255300 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚294616082867200 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚281413272777728 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19614852 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 _runtime‚3141.061255807 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5376530060288 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚296156699164346 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 trainer/global_step‚7649 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53282256 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58146851 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 135662521 _step‚618 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚227 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚286947314839040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚286947509670400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169927680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5534042061312 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚290796987096880 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58146849 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33667407 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚296156802352180 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5376530060288 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0ê‰|9›2¢- _wandbruntime‚3141  _runtime‚3141‚ gâ&£ÜM REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚286983909680128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77515425 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77515425 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5377296199168 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135669595 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5534916841472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚300030762472960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5360491206856 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚300030762472960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53289203 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5360405058270 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚294653466273792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58154172 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19617506 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19617256 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33671694 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3141.317854835 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚281448967934976 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚286983702273024 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚290833859164640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚294653449234432 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 _step‚619 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135679754 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 % _timestamp‚1.7845049872545137e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77525582 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58154170 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚300030657404416 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53288950 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33671697 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 trainer/global_step‚7649 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5377208169984 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5377208169984 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58154170 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚294653449234432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚296194247183550 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚281448992838656 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚294653466273792 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚296194350371496 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135679754 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77525582 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5534734338048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚290833842125280 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104ö]l¤áM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77515425 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚77515425 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5377296199168 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135669595 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5534916841472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚300030762472960 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5360491206856 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚300030762472960 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53289203 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5360405058270 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚294653466273792 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58154172 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19617506 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19617256 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33671694 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3141.317854835 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚281448967934976 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚286983702273024 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚290833859164640 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚294653449234432 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 _step‚619 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135679754 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 % _timestamp‚1.7845049872545137e+09 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚77525582 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 OADeviceStatsMonitor.on_train_bat9L¾{ch_end/requested_bytes.all.current‚ 103187946 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58154170 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚300030657404416 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53288950 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33671697 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 trainer/global_step‚7649 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5377208169984 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5377208169984 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58154170 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚294653449234432 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚296194247183550 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚281448992838656 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚294653466273792 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚296194350371496 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135679754 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77525582 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5534734338048 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚290833842125280 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚286983909680128 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330ër½S2¥- _wandbruntime‚3141  _runtime‚3141ÄÕ䦂 /loss/FBt1/detection_loss‚0.7205724552799685 /loss/FBt2/detection_loss‚0.5684573787139423 epoch‚9 % _timestamp‚1.7845049872560666e+09 /loss/FRt1/assignment_loss‚2.359476630572025 %loss/total_loss‚9.547673806887389 trainer/global_step‚7649 _step‚620 0loss/FRt2/assignment_loss‚0.8555110875103209 2loss/SRqqt2/assignment_loss‚0.6043051144285542 2loss/FBt2/assignment_loss‚0.004844023346084522 /loss/FRt2/detection_loss‚1.4053029770046355 1loss/SRqqt2/detection_loss‚0.6238195045280674 2loss/FBt1/assignment_loss‚0.014220592401305984 1loss/SRqqt1/detection_loss‚1.0663505847287642 _runtime‚3141.318902936 2loss/SRqqt1/assignment_loss‚0.7351699623531759 /loss/FRt1/detection_loss‚0.5896434960205444^2E §‡ 2loss/SRqqt1/assignment_loss‚0.7351699623531759 /loss/FRt1/detection_loss‚0.5896434960205444 /loss/FBt1/detection_loss‚0.7205724552799685 /loss/FBt2/detection_loss‚0.5684573787139423 epoch‚9 % _timestamp‚1.7845049872560666e+09 /loss/FRt1/assignment_loss‚2.359476630572025 %loss/total_loss‚9.547673806887389 trainer/global_step‚7649 _step‚620 0loss/FRt2/assignment_loss‚0.8555110875103209 2loss/SRqqt2/assignment_loss‚0.6043051144285542 2loss/FBt2/assignment_loss‚0.004844023346084522 /loss/FRt2/detection_loss‚1.4053029770046355 1loss/SRqqt2/detection_loss‚0.6238195045280674 2loss/FBt1/assignment_loss‚0.014220592401305984 1loss/SRqqt1/detection_loss‚1.0663505847287642 _runtime‚3141.318902936ìŸã*¤k¨‚(:à £ÅõÒàéì¿ network.sent‚ 1.0104243e+07% disk./.usageGB‚1165.7623329162598 disk.nvme0n1.in‚ 99.9921875% disk.nvme0n1.out‚17905.1337890625" disk.dm-0.out‚22494.2509765625 proc.memory.rssMB‚2873.75) proc.memory.percent‚17.53997802734375 proc.cpu.threads‚17) disk./.usagePercent‚75.93306190628626 disk.nvme0c0n1.in‚ 99.9921875 disk.nvme1n1.in‚ 10.48828125! disk.dm-1.out‚3062.4560546875' proc.memory.availableMB‚ 128.2890625& disk.nvme1c1n1.out‚7865.5888671875 disk.md0.in‚0 disk.md0.out‚ 100.59765625 disk.rbd0.in‚0 cpu‚26.468057405968732 network.recv‚900697' disk.nvme0c0n1.out‚17905.1337890625! disk.nvme1c1n1.in‚ 10.48828125$ disk.nvme1n1.out‚7865.5888671875 disk.dm-0.in‚ 110.48046875 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚99.21698570251465ß»±`©‚(:… £ÅõÒÍ–À gpu.0.gpu‚79.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚129.996' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚43.332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Èý/2ª- _wandbruntime‚3159  _runtime‚3159·b™É«Ã ' lr-AdamW/pg1‚0.00010379638174497831 ' lr-AdamW/pg2‚0.00010379638174497831 trainer/global_step‚7699 % _timestamp‚1.7845050057655258e+09 _runtime‚3159.829551673 _step‚621*I!7Î¬È % _timestamp‚1.7845050057655258e+09 _runtime‚3159.829551673 _step‚621 ' lr-AdamW/pg1‚0.00010379638174497831 ' lr-AdamW/pg2‚0.00010379638174497831 trainer/global_step‚7699í3DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5393644730708 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _step‚622 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚301894974092288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136525323 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚296484401228800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58512901 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58512899 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5410467802112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19738457 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚283198055543808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5410467802112 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚301894974092288 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚301894869030912 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚283198030640128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚7699 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5410555824128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5410555824128 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78002279 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58512901 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78012422 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚296484401228800 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚288767054552064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚296484418268160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 136515178 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚292640573445520 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚301894869030912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58512899 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78012422 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143_+±Ý'¯×O JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚301894869030912 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚283198030640128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚7699 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5410555824128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5410555824128 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78002279 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58512901 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78012422 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚296484401228800 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚288767054552064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚296484418268160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 136515178 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚292640573445520 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚301894869030912 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58512899 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78012422 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78002279 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5393644730708 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _step‚622 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚301894974092288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136525323 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚296484401228800 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58512901 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58512899 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5410467802112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19738457 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚283198055543808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5410467802112 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚301894974092288 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330î¼Õ2°- _wandbruntime‚3160  _runtime‚3160­Cöÿ±ÜM F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 _runtime‚3160.395599063 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19740698 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 % _timestamp‚1.7845050063322186e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78022334 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚296521784635392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33886197 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78022334 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5569725965824 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5411233940992 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58520220 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78012177 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚301933018576384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5569908469248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136542556 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚288803451763200 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5411145911808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33886194 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5394234533678 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5394320682264 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58520222 WCDeviceStatsMonitor.on_train_|ôÑ7ãbatch_end/active_bytes.large_pool.freed‚296521767596032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5411145911808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚296521767596032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚292677445513280 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5411233940992 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136532397 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7699 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚283233725797376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚283233750701056 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78012177 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚298071680046958 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53627154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19740957 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 _step‚623 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136542556 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58520220 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58520222 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53626892 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚298071783234904 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚288803659170304 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚292677462552640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482WÑT¤ç&²áM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58520222 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚296521767596032 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5411145911808 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚296521767596032 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚292677445513280 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5411233940992 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136532397 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚259 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7699 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚283233725797376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚283233750701056 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78012177 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚298071680046958 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53627154 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19740957 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 _step‚623 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136542556 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58520220 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58520222 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53626892 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚298071783234904 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚288803659170304 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚292677462552640 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 _runtime‚3160.395599063 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19740698 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 % _timestamp‚1.7845050063322186e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78022334 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚296521784635392 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚33886197 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78022334 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5569725965824 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5411233940992 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58520220 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78012177 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚301933018576384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5569908469248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136542556 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚262 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚288803451763200 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5411145911808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33886194 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5394234533678 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5394320682264ïÈçLH2³- _wandbruntime‚3160  _runtime‚3160ò¾Üý´€ 0loss/FRt1/assignment_loss‚2.1282021111226177 0loss/SRqqt1/detection_loss‚1.096712172240794 /loss/FBt1/detection_loss‚1.2582129829919295 %loss/total_loss‚9.769379082000777 trainer/global_step‚7699 2loss/SRqqt2/assignment_loss‚0.5350778503358096 1loss/FBt1/assignment_loss‚0.13347328470429012 /loss/FRt1/detection_loss‚0.6016348394010979 /loss/FRt2/detection_loss‚1.4000823529637303 /loss/FBt2/detection_loss‚0.5359354324011085 _runtime‚3160.396777325 1loss/FBt2/assignment_loss‚0.02859989443848756 epoch‚9 _step‚624 0loss/FRt2/assignment_loss‚0.8521376968907916 2loss/SRqqt1/assignment_loss‚0.5875007985784804 1loss/SRqqt2/detection_loss‚0.6118096659316384 % _timestamp‚1.7845050063337717e+09Ñ:]c µ… 0loss/SRqqt1/detection_loss‚1.096712172240794 /loss/FBt1/detection_loss‚1.2582129829919295 %loss/total_loss‚9.769379082000777 trainer/global_step‚7699 2loss/SRqqt2/assignment_loss‚0.5350778503358096 1loss/FBt1/assignment_loss‚0.13347328470429012 /loss/FRt1/detection_loss‚0.6016348394010979 /loss/FRt2/detection_loss‚1.4000823529637303 /loss/FBt2/detection_loss‚0.5359354324011085 _runtime‚3160.396777325 1loss/FBt2/assignment_loss‚0.02859989443848756 epoch‚9 _step‚624 0loss/FRt2/assignment_loss‚0.8521376968907916 2loss/SRqqt1/assignment_loss‚0.5875007985784804 1loss/SRqqt2/detection_loss‚0.6118096659316384 % _timestamp‚1.7845050063337717e+09 0loss/FRt1/assignment_loss‚2.1282021111226177ð×…·E^¶‚(:Ó ²Åõҥ߃Á network.recv‚905034) disk./.usagePercent‚75.93307209372074% disk./.usageGB‚1165.7624893188477 disk.rbd0.in‚0% disk.nvme0c0n1.out‚17906.86328125% disk.nvme1c1n1.out‚7866.431640625# disk.nvme1n1.out‚7866.431640625 disk.md0.in‚0! disk.nvme1c1n1.in‚ 10.48828125" disk.dm-0.out‚22495.4267578125! disk.dm-1.out‚3062.4951171875$ memory_percent‚99.21722412109375" proc.memory.availableMB‚128.25 proc.memory.rssMB‚2873.75) proc.memory.percent‚17.53997802734375 disk.rbd0.out‚0 disk.nvme1n1.in‚ 10.48828125 disk.dm-0.in‚ 110.48046875 proc.cpu.threads‚17 network.sent‚ 1.0153843e+07 disk.nvme0c0n1.in‚ 99.9921875 disk.nvme0n1.in‚ 99.9921875# disk.nvme0n1.out‚17906.86328125 disk.md0.out‚ 101.234375 disk.dm-1.in‚0 cpu‚26.465513817903485—oé·‚(:… ²ÅõÒ°¡ªÁ gpu.0.gpu‚76.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.684' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.228 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-“¤2¸- _wandbruntime‚3179  _runtime‚3179Ò2_ɹà ' lr-AdamW/pg1‚0.00010367831470340013 ' lr-AdamW/pg2‚0.00010367831470340013 trainer/global_step‚7749 % _timestamp‚1.7845050249462655e+09 _runtime‚3179.010592606 _step‚6251‹Ñ$ÎºÈ ' lr-AdamW/pg2‚0.00010367831470340013 trainer/global_step‚7749 % _timestamp‚1.7845050249462655e+09 _runtime‚3179.010592606 _step‚625 ' lr-AdamW/pg1‚0.00010367831470340013ñYm î2»- _wandbruntime‚3179  _runtime‚31798í=æØ'¼ÒO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5444493565952 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58878951 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚298352736629760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34096407 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚78509174 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5444405543936 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5604186512384 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78499031 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53957692 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53957447 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34096404 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5604001904640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5444493565952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚299911564891162 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚303797230195712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚298352736629760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚294484193872880 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5444405543936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚284982788502528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚626 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚290586999918592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58878949 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78499031 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚7749 _runtime‚3179.010681136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚290586790407168 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 137377980 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58878951 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚298352719590400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78509174 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚294484176833520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚298352719590400 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19861285 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚284982813406208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚303797125134336 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚303797125134336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58878949 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚290586790407168 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 137377980 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58878951 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚298352719590400 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78509174 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚294484176833520 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚298352719590400 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19861285 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚284982813406208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚303797125134336 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚303797125134336 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚58878949 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5604186512384 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚78499031 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚53957692 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53957447 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34096404 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5604001904640 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5444493565952 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚299911564891162 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚303797230195712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚298352736629760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚294484193872880 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5444405543936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚284982788502528 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚626 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚290586999918592 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58878949 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78499031 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚7749 _runtime‚3179.010681136 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53964160 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19863732 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78519086 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5445171682816 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 137395199 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚298390085957632 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 $ _timestamp‚1.784505025208996e+09 _step‚627 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53964429 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚303835274679808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34100697 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚299949112910366 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137405358 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 137405358 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚290623394214400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚294521065940640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚303835169611264 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚298390102996992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚299949216098312 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58886272 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78508929 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58886270 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58886270 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚3179.272242808 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78519086 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34100694 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5604885650944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5445171682816 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚285018483659776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5428150157672 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚7749 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚298390102996992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚303835274679808 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58886272 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 137395199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚298390085957632 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚294521048901280 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5445083653632 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19863466 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚285018508563456 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5604698953216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5428064009086 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2J}Žæ&ÀàM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58886272 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 137395199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚298390085957632 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚294521048901280 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5445083653632 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19863466 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚285018508563456 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5604698953216 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5428064009086 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53964160 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19863732 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚78519086 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5445171682816 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 137395199 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚298390085957632 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 $ _timestamp‚1.784505025208996e+09 _step‚627 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚53964429 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚303835274679808 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34100697 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚299949112910366 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 137405358 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 137405358 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚290623394214400 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚294521065940640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚303835169611264 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚298390102996992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚299949216098312 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58886272 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78508929 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58886270 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58886270 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚3179.272242808 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚78519086 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34100694 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5604885650944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5445171682816 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚285018483659776 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5428150157672 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚7749 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚298390102996992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚303835274679808óÑAñ÷2Á- _wandbruntime‚3179  _runtime‚3179EjΠ2loss/SRqqt2/assignment_loss‚0.6755932572232268 1loss/SRqqt2/detection_loss‚0.7192988229224284 &loss/total_loss‚10.066929595588975 % _timestamp‚1.7845050252099738e+09 _runtime‚3179.272815139 _step‚628 1loss/FBt1/assignment_loss‚0.03354767742457751 2loss/FBt2/assignment_loss‚0.008074883768009525 .loss/FBt1/detection_loss‚0.733985161408782 epoch‚9 0loss/FRt1/assignment_loss‚2.3275605793810485 /loss/FRt1/detection_loss‚0.5359639172895405 0loss/SRqqt1/detection_loss‚1.255884552095765 /loss/FBt2/detection_loss‚0.4374382501729767 0loss/FRt2/assignment_loss‚1.1476729741053922 2loss/SRqqt1/assignment_loss‚0.6391390401557305 /loss/FRt2/detection_loss‚1.5527704796414972 trainer/global_step‚7749ŽÓ Æ 2loss/SRqqt2/assignment_loss‚0.6755932572232268 1loss/SRqqt2/detection_loss‚0.7192988229224284 &loss/total_loss‚10.066929595588975 % _timestamp‚1.7845050252099738e+09 _runtime‚3179.272815139 _step‚628 1loss/FBt1/assignment_loss‚0.03354767742457751 2loss/FBt2/assignment_loss‚0.008074883768009525 .loss/FBt1/detection_loss‚0.733985161408782 epoch‚9 0loss/FRt1/assignment_loss‚2.3275605793810485 /loss/FRt1/detection_loss‚0.5359639172895405 0loss/SRqqt1/detection_loss‚1.255884552095765 /loss/FBt2/detection_loss‚0.4374382501729767 0loss/FRt2/assignment_loss‚1.1476729741053922 2loss/SRqqt1/assignment_loss‚0.6391390401557305 /loss/FRt2/detection_loss‚1.5527704796414972 trainer/global_step‚7749ô ³4ÔdÄ‚(:Ù ÁÅõÒҸѿ network.recv‚909343) disk./.usagePercent‚75.92041085197724 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 99.9921875( proc.memory.availableMB‚ 127.99609375 disk.rbd0.in‚0( disk.nvme0c0n1.out‚17932.19970703125 disk.nvme0n1.in‚ 99.9921875% disk.nvme1c1n1.out‚7867.208984375 proc.memory.rssMB‚2873.75% disk./.usageGB‚1165.5681076049805# disk.dm-0.out‚22520.40478515625 disk.rbd0.out‚0& disk.nvme0n1.out‚17932.19970703125! disk.nvme1c1n1.in‚ 10.48828125 disk.nvme1n1.in‚ 10.48828125# disk.nvme1n1.out‚7867.208984375 disk.md0.out‚ 101.64453125 network.sent‚ 1.02161e+07 disk.dm-0.in‚ 110.48046875 disk.dm-1.out‚ 3062.734375 disk.md0.in‚0$ memory_percent‚99.21877384185791) proc.memory.percent‚17.53997802734375 cpu‚26.463273667198 proc.cpu.threads‚17*TkW›Å‚(: ÁÅõÒëÅ¢À gpu.0.gpu‚16.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚112.049' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.34966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èÈû]bÆ‚(:× ÐÅõÒÕÅ¿ cpu‚26.460953971916204 network.sent‚ 1.0279474e+07) disk./.usagePercent‚75.94425789678239 disk.md0.in‚0 disk.dm-1.out‚ 3062.88671875 proc.memory.rssMB‚2873.75 proc.cpu.threads‚17 network.recv‚913498% disk./.usageGB‚1165.9342193603516" disk.dm-0.out‚22583.4462890625& disk.nvme1c1n1.out‚7868.5595703125) proc.memory.percent‚17.53997802734375 disk.md0.out‚ 102.46875 disk.rbd0.in‚0& disk.nvme0c0n1.out‚17995.767578125 disk.nvme0n1.in‚ 99.9921875$ disk.nvme0n1.out‚17995.767578125 disk.nvme1n1.in‚ 10.48828125$ disk.nvme1n1.out‚7868.5595703125 disk.dm-0.in‚ 110.48046875 disk.dm-1.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 99.9921875! disk.nvme1c1n1.in‚ 10.48828125$ memory_percent‚99.21894073486328% proc.memory.availableMB‚ 127.96875@öiœÇ‚(:‘ ÐÅõÒÅ©É¿ gpu.0.gpu‚60.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚164.849' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚54.949666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02)b†2È- _wandbruntime‚3198  _runtime‚3198l­¥ÉÉà % _timestamp‚1.7845050441138294e+09 _runtime‚3198.176968266 _step‚629 ' lr-AdamW/pg1‚0.00010355948501404904 ' lr-AdamW/pg2‚0.00010355948501404904 trainer/global_step‚7799¤.øÒÎÊÈ trainer/global_step‚7799 % _timestamp‚1.7845050441138294e+09 _runtime‚3198.176968266 _step‚629 ' lr-AdamW/pg1‚0.00010355948501404904 ' lr-AdamW/pg2‚0.00010355948501404904õF^ûÌ2Ë- _wandbruntime‚3198  _runtime‚3198(-»aTÌÒO % _timestamp‚1.7845050441147542e+09 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5478343285760 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5461303681524 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59244999 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚286767546364928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚296327797260880 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54294339 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚305699381237760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79005926 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5478343285760 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚300221054991360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79005926 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚301788997754570 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54294339 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚305699381237760 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79005926 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5478343285760 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚300221054991360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79005926 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚301788997754570 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5478343285760 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5461303681524 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59244999 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚286767546364928 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚296327797260880 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5639686583808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 _step‚631 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚305737530783232 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚300258404319232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79005681 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5639866990080 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34315197 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59252320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59252322 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54301334 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79005681 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138258001 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚286803266425856 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138258001 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚300258421358592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138268160 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚292443133415936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5461979633080 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19985881 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚305737530783232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19986137 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚301826545773774 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59252320 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5479021395456 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚300258404319232 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79015838 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138268160 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚292442928105984 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5461893484494 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚305737425714688 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79015838 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5479021395456 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54301075 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138268160 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚292442928105984 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5461893484494 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚305737425714688 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79015838 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5479021395456 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54301075 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5639686583808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 _step‚631 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚305737530783232 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚300258404319232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79005681 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5639866990080 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34315197 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59252320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚259 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59252322 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54301334 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79005681 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138258001 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚286803266425856 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138258001 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚300258421358592 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138268160 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚292443133415936 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5461979633080 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19985881 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚305737530783232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19986137 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482÷B·b02Ñ- _wandbruntime‚3198  _runtime‚3198ïì&]Ò‚ &loss/total_loss‚10.004032043920372 epoch‚9 /loss/FRt2/assignment_loss‚1.092635011413823 /loss/FBt1/detection_loss‚0.8479847987744051 /loss/FBt2/detection_loss‚0.6002383138428429 trainer/global_step‚7799 % _timestamp‚1.7845050443788328e+09 _step‚632 /loss/FRt1/assignment_loss‚2.283248770162772 2loss/SRqqt1/assignment_loss‚0.5737276183535869 2loss/SRqqt2/assignment_loss‚0.6462692462588603 2loss/FBt1/assignment_loss‚0.004915366534802956 2loss/FBt2/assignment_loss‚0.014761089189178522 /loss/FRt1/detection_loss‚0.6047057550899034 _runtime‚3198.442204038 /loss/FRt2/detection_loss‚1.5106729856692254 1loss/SRqqt1/detection_loss‚1.0888250009156764 1loss/SRqqt2/detection_loss‚0.7360480877152942gá Ó‡ /loss/FRt2/detection_loss‚1.5106729856692254 1loss/SRqqt1/detection_loss‚1.0888250009156764 1loss/SRqqt2/detection_loss‚0.7360480877152942 &loss/total_loss‚10.004032043920372 epoch‚9 /loss/FRt2/assignment_loss‚1.092635011413823 /loss/FBt1/detection_loss‚0.8479847987744051 /loss/FBt2/detection_loss‚0.6002383138428429 trainer/global_step‚7799 % _timestamp‚1.7845050443788328e+09 _step‚632 /loss/FRt1/assignment_loss‚2.283248770162772 2loss/SRqqt1/assignment_loss‚0.5737276183535869 2loss/SRqqt2/assignment_loss‚0.6462692462588603 2loss/FBt1/assignment_loss‚0.004915366534802956 2loss/FBt2/assignment_loss‚0.014761089189178522 /loss/FRt1/detection_loss‚0.6047057550899034 _runtime‚3198.442204038ø9t…kÔ‚(:à ßÅõÒ×®ú¿( disk./.usagePercent‚75.6923753185238" disk.nvme1n1.out‚ 7904.98828125 disk.md0.out‚ 104.7890625 disk.dm-1.in‚0( disk.nvme0c0n1.out‚18193.23779296875 proc.memory.rssMB‚2873.75 network.sent‚ 1.0296139e+07% disk./.usageGB‚1162.0671920776367 disk.dm-0.in‚ 110.5078125! disk.dm-1.out‚3063.6787109375 disk.rbd0.out‚0& disk.nvme0n1.out‚18193.23779296875) proc.memory.percent‚17.53997802734375 network.recv‚917057# disk.dm-0.out‚22811.83447265625 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 100.01953125 disk.nvme0n1.in‚ 100.01953125! disk.nvme1c1n1.in‚ 10.48828125$ disk.nvme1c1n1.out‚ 7904.98828125( proc.memory.availableMB‚ 128.14453125 disk.nvme1n1.in‚ 10.48828125 disk.md0.in‚0$ memory_percent‚99.21786785125732 cpu‚26.45864711828293 proc.cpu.threads‚17âç"£›Õ‚(: ßÅõÒ¤®¥À gpu.0.gpu‚60.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚194.282' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚64.76066666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n‹=2Ö- _wandbruntime‚3217  _runtime‚3217É°…0É×à trainer/global_step‚7849 % _timestamp‚1.7845050632789407e+09 _runtime‚3217.342066267 _step‚633 ' lr-AdamW/pg1‚0.00010343989458022374 ' lr-AdamW/pg2‚0.00010343989458022374~!ÎØÈ _step‚633 ' lr-AdamW/pg1‚0.00010343989458022374 ' lr-AdamW/pg2‚0.00010343989458022374 trainer/global_step‚7849 % _timestamp‚1.7845050632789407e+09 _runtime‚3217.342066267ù‹'°«2Ù- _wandbruntime‚3217  _runtime‚3217hq§vØ'ÚÒO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79502678 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54632665 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79492535 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚302089373352960 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59611049 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚298171383609520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 139113729 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34525407 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚288552304227328 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5512369049600 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79492535 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59611049 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54632424 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚288552329131008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5674151911936 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5512369049600 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20107258 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5673975692800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20107020 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845050632798424e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79502678 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚307601637341184 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302089373352960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5495133156932 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5495047008458 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 139103584 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚634 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚294226481042944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201122816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚302089356313600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34525404 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362€4ÿÝ'Û×O YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚302089356313600 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34525404 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79502678 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54632665 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79492535 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚302089373352960 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59611049 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚298171383609520 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 139113729 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34525407 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚288552304227328 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5512369049600 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79492535 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59611049 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54632424 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚288552329131008 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5674151911936 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5512369049600 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20107258 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5673975692800 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20107020 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845050632798424e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79502678 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚307601637341184 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302089373352960 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5495133156932 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5495047008458 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 139103584 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚634 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚294226481042944 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201122816ú¾t2Ü- _wandbruntime‚3217  _runtime‚3217)ê ìòÝÜM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7845050635397267e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79512590 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚288588024288256 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚288587999384576 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚294262671089664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 trainer/global_step‚7849 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚302126739720192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59618370 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59618372 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79502433 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298208255677280 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5495722959902 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool0}üð.peak‚152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79512590 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34529694 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59618370 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚307639786886656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚302126722680832 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5512959137280 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34529697 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚303704081825128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚302126739720192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79502433 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5513047166464 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139130962 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚635 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚302126722680832 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5512959137280 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 139120803 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20109651 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚294262880593920 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚303703978637182 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 139120803 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54639096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680s‚Yºç&ÞáM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚303703978637182 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 139120803 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54639096 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7845050635397267e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79512590 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚288588024288256 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚288587999384576 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚294262671089664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 trainer/global_step‚7849 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚302126739720192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59618370 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59618372 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79502433 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298208255677280 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5495722959902 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79512590 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34529694 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59618370 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚307639786886656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚302126722680832 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5512959137280 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34529697 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚303704081825128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚302126739720192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79502433 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5513047166464 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139130962 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚635 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚302126722680832 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5512959137280 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 139120803 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20109651 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚294262880593920ûcÉ‚>2ß- _wandbruntime‚3217  _runtime‚3217Bè=Gàû 2loss/SRqqt2/assignment_loss‚0.5712969180071154 1loss/SRqqt2/detection_loss‚0.6719367419949804 epoch‚9 _runtime‚ 3217.604357 /loss/FRt1/assignment_loss‚2.237467173501557 0loss/FRt2/assignment_loss‚0.9144550005035221 2loss/SRqqt1/assignment_loss‚0.6040696992748417 1loss/FBt2/assignment_loss‚0.04185665792416677 /loss/FRt1/detection_loss‚0.5471471208781141 trainer/global_step‚7849 $ _timestamp‚1.784505063541379e+09 _step‚636 .loss/FRt2/detection_loss‚1.619766083161211 .loss/FBt1/detection_loss‚2.060958923655562 3loss/FBt1/assignment_loss‚0.0011400932354490578 0loss/SRqqt1/detection_loss‚0.990941895203044 .loss/FBt2/detection_loss‚0.364990644818662 &loss/total_loss‚10.626026952158227ËJWဠ&loss/total_loss‚10.626026952158227 2loss/SRqqt2/assignment_loss‚0.5712969180071154 1loss/SRqqt2/detection_loss‚0.6719367419949804 epoch‚9 _runtime‚ 3217.604357 /loss/FRt1/assignment_loss‚2.237467173501557 0loss/FRt2/assignment_loss‚0.9144550005035221 2loss/SRqqt1/assignment_loss‚0.6040696992748417 1loss/FBt2/assignment_loss‚0.04185665792416677 /loss/FRt1/detection_loss‚0.5471471208781141 trainer/global_step‚7849 $ _timestamp‚1.784505063541379e+09 _step‚636 .loss/FRt2/detection_loss‚1.619766083161211 .loss/FBt1/detection_loss‚2.060958923655562 3loss/FBt1/assignment_loss‚0.0011400932354490578 0loss/SRqqt1/detection_loss‚0.990941895203044 .loss/FBt2/detection_loss‚0.364990644818662üÄI®iâ‚(:Þ îÅõÒŸ—ã¿) disk./.usagePercent‚75.76305101795629( disk.nvme0c0n1.out‚18489.67333984375$ disk.nvme1c1n1.out‚ 7907.48828125 disk.nvme1n1.in‚ 10.48828125 disk.md0.in‚0# disk.dm-0.out‚23109.72705078125# memory_percent‚99.2180585861206) proc.memory.percent‚17.53997802734375% disk./.usageGB‚1163.1522407531738" disk.nvme0c0n1.in‚ 100.40234375 disk.nvme0n1.in‚ 100.40234375& disk.nvme0n1.out‚18489.67333984375! disk.nvme1c1n1.in‚ 10.48828125 disk.md0.out‚ 105.234375 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚920482 disk.dm-0.in‚ 110.890625 disk.dm-1.in‚0( proc.memory.availableMB‚ 128.11328125 proc.memory.rssMB‚2873.75 cpu‚26.45621753192992 proc.cpu.threads‚17 network.sent‚ 1.0344578e+07" disk.nvme1n1.out‚ 7907.48828125! disk.dm-1.out‚3063.7763671875mº œã‚(:‘ îÅõҺȷÀ gpu.0.gpu‚42.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.807' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.269000000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àÙßc2ä- _wandbruntime‚3236  _runtime‚3236ÍÊÉåà ' lr-AdamW/pg1‚0.00010331954531740772 ' lr-AdamW/pg2‚0.00010331954531740772 trainer/global_step‚7899 % _timestamp‚1.7845050823657355e+09 _runtime‚3236.429575822 _step‚637·'ÒÃÎæÈ ' lr-AdamW/pg1‚0.00010331954531740772 ' lr-AdamW/pg2‚0.00010331954531740772 trainer/global_step‚7899 % _timestamp‚1.7845050823657355e+09 _runtime‚3236.429575822 _step‚637ý”®F)2ç- _wandbruntime‚3236  _runtime‚3236‘fˆn×'èÑO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54969322 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79989287 trainer/global_step‚7899 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 139976531 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59977101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20229679 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79999430 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79989287 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34739904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚290337062089728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚3236.429644682 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5709160736768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚305543863481386 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚300015004036880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54969586 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚309503998505984 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚296046038218752 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚296046247730176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚290337086993408 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5708976129024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 $ _timestamp‚1.784505082366475e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59977099 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59977099 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚303957691714560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚300014986997520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5528962632340 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5528876483866 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5546306791424 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5546218769408 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚309503893444608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚309503998505984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5546306791424 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚303957691714560 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482YÑ} ¾éÖO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚303957691714560 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54969322 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79989287 trainer/global_step‚7899 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 139976531 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚59977101 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20229679 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 QCDeviceStatsMonitor.on_train_baÌ ¯"tch_start/requested_bytes.all.current‚ 103187834 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79999430 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79989287 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34739904 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚290337062089728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚3236.429644682 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5709160736768 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚305543863481386 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚300015004036880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54969586 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚309503998505984 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚296046038218752 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚296046247730176 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚290337086993408 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5708976129024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 $ _timestamp‚1.784505082366475e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59977099 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59977099 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚303957691714560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚300014986997520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5528962632340 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5528876483866 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5546306791424 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5546218769408 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚309503893444608 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚309503998505984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5546306791424 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751þZ W2ê- _wandbruntime‚3236  _runtime‚3236Í|õ*â&ëÜM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5709678932480 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 139993764 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚303995058081792 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚290372782150656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚7899 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79999185 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 % _timestamp‚1.7845050826289694e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54976123 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚300051876104640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5529638583896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚290372757246976 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5529552435310 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚296082436179456 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59984420 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59984420 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79999185 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚300051859065280 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139993764 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 139983605 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54976363 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚305581411500590 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59984422 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5546896879104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5546984908288 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34744197 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20231929 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚305581514688536 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚303995058081792 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20232166 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544Û#‰hç&ìáM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54976123 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚300051876104640 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5529638583896 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚290372757246976 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5529552435310 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚296082436179456 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59984420 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59984420 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79999185 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚300051859065280 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 139993764 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 139983605 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54976363 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚305581411500590 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59984422 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5546896879104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5546984908288 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34744197 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20231929 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚305581514688536 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚303995058081792 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20232166 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5709678932480 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 139993764 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚303995058081792 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚290372782150656 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚7899 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚79999185 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 % _timestamp‚1.7845050826289694e+09ÿí›`M2í- _wandbruntime‚3236  _runtime‚3236éãù î€ /loss/FRt2/assignment_loss‚1.058233326412292 _runtime‚3236.692757847 _step‚640 2loss/SRqqt1/assignment_loss‚0.6146811064025872 /loss/FBt1/detection_loss‚0.8869228487213453 %loss/total_loss‚9.881174380443529 trainer/global_step‚7899 2loss/SRqqt2/assignment_loss‚0.6686302234846697 /loss/FRt1/detection_loss‚0.5909956623647272 0loss/SRqqt1/detection_loss‚1.132459688415235 1loss/SRqqt2/detection_loss‚0.6654758997452307 epoch‚9 % _timestamp‚1.7845050826299436e+09 /loss/FRt1/assignment_loss‚2.342790042360624 2loss/FBt1/assignment_loss‚0.023460584153780332 2loss/FBt2/assignment_loss‚0.008043732601304216 /loss/FRt2/detection_loss‚1.4134242205706191 /loss/FBt2/detection_loss‚0.4760570452111124QáE ï… % _timestamp‚1.7845050826299436e+09 /loss/FRt1/assignment_loss‚2.342790042360624 2loss/FBt1/assignment_loss‚0.023460584153780332 2loss/FBt2/assignment_loss‚0.008043732601304216 /loss/FRt2/detection_loss‚1.4134242205706191 /loss/FBt2/detection_loss‚0.4760570452111124 /loss/FRt2/assignment_loss‚1.058233326412292 _runtime‚3236.692757847 _step‚640 2loss/SRqqt1/assignment_loss‚0.6146811064025872 /loss/FBt1/detection_loss‚0.8869228487213453 %loss/total_loss‚9.881174380443529 trainer/global_step‚7899 2loss/SRqqt2/assignment_loss‚0.6686302234846697 /loss/FRt1/detection_loss‚0.5909956623647272 0loss/SRqqt1/detection_loss‚1.132459688415235 1loss/SRqqt2/detection_loss‚0.6654758997452307 epoch‚9€ŽÝ®Woð‚(:ä ýÅõÒµô¿ proc.cpu.threads‚17 network.sent‚ 1.0407965e+07) disk./.usagePercent‚75.58586469711456% disk.nvme1n1.out‚7908.54736328125" disk.dm-1.out‚3063.91845703125 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 104.47265625 proc.memory.rssMB‚2873.75 disk.md0.in‚0 disk.dm-0.in‚ 114.9609375 disk.dm-1.in‚0' disk.nvme0c0n1.out‚19575.1845703125% disk.nvme0n1.out‚19575.1845703125 disk.rbd0.out‚0% disk./.usageGB‚1160.4319877624512 disk.nvme1n1.in‚ 10.48828125" disk.dm-0.out‚24194.5791015625 disk.nvme0n1.in‚ 104.47265625! disk.nvme1c1n1.in‚ 10.48828125 network.recv‚924392 disk.md0.out‚ 105.984375' disk.nvme1c1n1.out‚7908.54736328125# memory_percent‚99.2182970046997( proc.memory.availableMB‚ 128.07421875) proc.memory.percent‚17.53997802734375 cpu‚26.454265877974912Çù¼›ñ‚(: ýÅõÒúàÎÀ gpu.0.gpu‚12.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚173.503' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚57.83433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0•^õê2ò- _wandbruntime‚3254  _runtime‚3254¦Z·Éóà ' lr-AdamW/pg1‚0.00010319843915323874 ' lr-AdamW/pg2‚0.00010319843915323874 trainer/global_step‚7949 % _timestamp‚1.7845051004946256e+09 _runtime‚3254.557743402 _step‚641z<Å!ÎôÈ % _timestamp‚1.7845051004946256e+09 _runtime‚3254.557743402 _step‚641 ' lr-AdamW/pg1‚0.00010319843915323874 ' lr-AdamW/pg2‚0.00010319843915323874 trainer/global_step‚7949nŸŸð2õ- _wandbruntime‚3254  _runtime‚3254£¿‘Þ=öÒO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚305825993036800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140829188 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5580244533248 UFDeviceStatsMonitor.on_train_batch_start/reserved_byt‚qS›%es.large_pool.peak‚ 6618611712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55307049 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20352896 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60343149 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚297865806987264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚642 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5744167448576 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34954404 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚292121819952128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5562792107748 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60343149 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚297866012304384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚307421399532628 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5580156511232 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚311406149548032 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚305825993036800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚292121844855808 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80486039 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 % _timestamp‚1.7845051004954197e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5580156511232 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34954407 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20352645 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚307421296344794 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5580244533248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152ƒ7¿"Ý'÷×O GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚7949 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚305826010076160 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5744167448576 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34954404 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚292121819952128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5562792107748 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60343149 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚297866012304384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚307421399532628 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5580156511232 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚311406149548032 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚305825993036800 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚292121844855808 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80486039 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 % _timestamp‚1.7845051004954197e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5580156511232 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34954407 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20352645 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚307421296344794 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5580244533248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚305825993036800 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 140829188 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5580244533248 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55307049 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20352896 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60343149 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚297865806987264 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚642 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80495937 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5580834620928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80495937 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60350472 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55313661 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20355216 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚292157540013056 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚297902203362304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60350470 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34958694 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5744688252928 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚301895479492640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚311444194024960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34958697 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚297902410769408 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚292157515109376 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5563468059304 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20354967 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 _runtime‚3254.809989958 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60350472 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚305863376443392 % _timestamp‚1.7845051007467015e+09 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140846407 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚307458947551944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5580922650112 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80506094 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140856566 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159¾cd˜Q úáM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140846407 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚307458947551944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E–318767104 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80506094 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚301895462453280 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55313913 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚307458844363998 _step‚643 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5580834620928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5580922650112 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140846407 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5744870756352 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5580922650112 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚80506094 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140856566 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚311444299093504 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚305863359404032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80495937 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5580834620928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80495937 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60350472 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55313661 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20355216 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚292157540013056 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚297902203362304 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60350470 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34958694 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5744688252928 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚301895479492640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚311444194024960 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34958697 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚297902410769408 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚292157515109376 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5563468059304 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20354967 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 _runtime‚3254.809989958 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60350472 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚305863376443392 % _timestamp‚1.7845051007467015e+09 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0ƒ˜JÄ2û- _wandbruntime‚3254  _runtime‚3254»Ãþüþ 2loss/SRqqt1/assignment_loss‚0.6996468099036052 1loss/SRqqt1/detection_loss‚1.0938896917985759 _runtime‚3254.810490378 0loss/FBt2/detection_loss‚0.49801826803634563 epoch‚9 trainer/global_step‚7949 _step‚644 /loss/FRt1/assignment_loss‚2.410223446675201 2loss/SRqqt2/assignment_loss‚0.6701879303366941 /loss/FRt1/detection_loss‚0.6131156111829746 /loss/FRt2/detection_loss‚1.4561278682827723 /loss/FBt1/detection_loss‚1.0551605040414467 % _timestamp‚1.7845051007475963e+09 /loss/FRt2/assignment_loss‚1.075371237884041 .loss/FBt1/assignment_loss‚0.11926484472044 1loss/FBt2/assignment_loss‚0.07909285960309945 1loss/SRqqt2/detection_loss‚0.6331475625463002 &loss/total_loss‚10.403246635011495Äí› ýƒ &loss/total_loss‚10.403246635011495 2loss/SRqqt1/assignment_loss‚0.6996468099036052 1loss/SRqqt1/detection_loss‚1.0938896917985759 _runtime‚3254.810490378 0loss/FBt2/detection_loss‚0.49801826803634563 epoch‚9 trainer/global_step‚7949 _step‚644 /loss/FRt1/assignment_loss‚2.410223446675201 2loss/SRqqt2/assignment_loss‚0.6701879303366941 /loss/FRt1/detection_loss‚0.6131156111829746 /loss/FRt2/detection_loss‚1.4561278682827723 /loss/FBt1/detection_loss‚1.0551605040414467 % _timestamp‚1.7845051007475963e+09 /loss/FRt2/assignment_loss‚1.075371237884041 .loss/FBt1/assignment_loss‚0.11926484472044 1loss/FBt2/assignment_loss‚0.07909285960309945 1loss/SRqqt2/detection_loss‚0.6331475625463002„"”"Bœþ‚(:‘ ŒÆõҎз¿ gpu.0.gpu‚63.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚112.239' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.413000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06bÜ‚pÿ‚(:å ŒÆõҨǻÀ# disk.dm-0.out‚24217.22216796875 network.recv‚928733" disk.nvme0c0n1.in‚ 104.47265625$ memory_percent‚99.21989440917969$ proc.memory.availableMB‚127.8125 proc.memory.rssMB‚2873.75 proc.cpu.threads‚17' disk.nvme1c1n1.out‚7912.11083984375 disk.rbd0.out‚0 disk.nvme0n1.in‚ 104.47265625& disk.nvme0n1.out‚19595.30908203125! disk.nvme1c1n1.in‚ 10.48828125 disk.md0.out‚ 106.44140625) disk./.usagePercent‚75.58585177646594 disk.rbd0.in‚0( disk.nvme0c0n1.out‚19595.30908203125 disk.dm-0.in‚ 114.9609375) proc.memory.percent‚17.53997802734375 cpu‚26.45232504668846 network.sent‚ 1.0469071e+07% disk./.usageGB‚1160.4317893981934 disk.dm-1.in‚0" disk.dm-1.out‚3063.98486328125 disk.nvme1n1.in‚ 10.48828125% disk.nvme1n1.out‚7912.11083984375 disk.md0.in‚0¾>En€‚(:ã ›ÆõÒ¡´í¿ disk.dm-1.in‚0( proc.memory.availableMB‚ 127.70703125) proc.memory.percent‚17.53997802734375 network.sent‚ 1.0531534e+07" disk.nvme0c0n1.in‚ 106.19140625! disk.nvme1c1n1.in‚ 10.48828125 disk.md0.in‚0 disk.rbd0.in‚0 proc.memory.rssMB‚2873.75 network.recv‚932785 disk.nvme0n1.in‚ 106.19140625' disk.nvme1c1n1.out‚7913.20361328125 disk.nvme1n1.in‚ 10.48828125# disk.dm-0.out‚24224.92822265625 disk.dm-1.out‚ 3064.1015625$ memory_percent‚99.22053813934326 proc.cpu.threads‚17) disk./.usagePercent‚75.58582469279865% disk.nvme1n1.out‚7913.20361328125 disk.md0.out‚ 107.30078125 disk.rbd0.out‚0 cpu‚26.45026932807189% disk./.usageGB‚1160.4313735961914' disk.nvme0c0n1.out‚19603.8193359375% disk.nvme0n1.out‚19603.8193359375 disk.dm-0.in‚ 116.6796875†»Š‘‚(:† ›ÆõÒ¸ë«À gpu.0.gpu‚82.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚115.131' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.377 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§ð+2‚- _wandbruntime‚3272  _runtime‚3272ãàÉɃà ' lr-AdamW/pg1‚0.00010307657802747784 ' lr-AdamW/pg2‚0.00010307657802747784 trainer/global_step‚7999 % _timestamp‚1.7845051187710738e+09 _runtime‚3272.835605874 _step‚645À5rÎ„È % _timestamp‚1.7845051187710738e+09 _runtime‚3272.835605874 _step‚645 ' lr-AdamW/pg1‚0.00010307657802747784 ' lr-AdamW/pg2‚0.00010307657802747784 trainer/global_step‚7999…\1A 2…- _wandbruntime‚3272  _runtime‚3272LÀC?Ø'†ÒO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚307694311398400 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55644698 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55644433 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60709199 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚293906577814528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5778970146816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚307694328437760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35168904 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚299685547961344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚303702193773520 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141691990 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚293906602718208 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚309298729208202 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5596621583156 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚313308405651456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60709201 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚303702210812880 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80982791 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35168907 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5614182275072 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199025664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330º@Ý'‡×O N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60709201 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚309298832396036 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5614182275072 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20475529 trainer/global_step‚7999 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5614182275072 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199025664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚307694311398400 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55644698 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55644433 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚60709199 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚293906577814528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5778970146816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚307694328437760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35168904 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚299685547961344 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚303702193773520 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 141691990 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚293906602718208 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚309298729208202 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5596621583156 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚313308405651456 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60709201 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚303702210812880 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80982791 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35168907 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.oj,hû%n_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141719368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60716520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚307731677765632 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚313346555196928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 _runtime‚3273.085201089 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5597297534712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141709209 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚309336277227406 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20478218 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5614772362752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20477960 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5779662910976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚307731694804992 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚303739065841280 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60716522 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚647 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚307731694804992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5614860391936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141709209 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚299721935882752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚309336380415352 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚7999 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5614860391936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35173197 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5614772362752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55651415 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚299722145387008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60716520 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚293942272971776 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80992689 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80992689 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5597211386126 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚313346450128384 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55651154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 $ _timestamp‚1.784505119021879e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚307731677765632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60716522 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35173194 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464Åÿ·€æ&ŠàM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚309336277227406 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20478218 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5614772362752 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚258 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20477960 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5779662910976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚307731694804992 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚303739065841280 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60716522 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚647 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚307731694804992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5614860391936 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 141709209 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚299721935882752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚309336380415352 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚7999 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5614860391936 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35173197 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5614772362752 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55651415 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚299722145387008 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60716520 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚293942272971776 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚80992689 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚80992689 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5597211386126 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚313346450128384 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55651154 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 $ _timestamp‚1.784505119021879e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚307731677765632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60716522 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35173194 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5779847511552 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141719368 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60716520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚307731677765632 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚313346555196928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 _runtime‚3273.085201089 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚261 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5597297534712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141709209 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0‡z¹ô-2‹- _wandbruntime‚3273  _runtime‚3273„_I Œƒ _step‚648 0loss/FRt2/assignment_loss‚0.9400670532876633 1loss/SRqqt2/detection_loss‚0.5728531583396506 /loss/FBt1/detection_loss‚1.0245867256252539 /loss/FRt1/assignment_loss‚2.329081761258064 /loss/FRt1/detection_loss‚0.5372779953425562 1loss/SRqqt1/detection_loss‚0.9598945630328922 trainer/global_step‚7999 2loss/SRqqt1/assignment_loss‚0.6876888892645466 3loss/FBt2/assignment_loss‚0.0005278553941520449 /loss/FBt2/detection_loss‚0.5592180362933117 _runtime‚ 3273.08576764 2loss/SRqqt2/assignment_loss‚0.6080657393120059 3loss/FBt1/assignment_loss‚0.0032085538011409924 /loss/FRt2/detection_loss‚1.4173989803757003 %loss/total_loss‚9.639869311326937 epoch‚9 % _timestamp‚1.7845051190228467e+09pÍÙˆ epoch‚9 % _timestamp‚1.7845051190228467e+09 _step‚648 0loss/FRt2/assignment_loss‚0.9400670532876633 1loss/SRqqt2/detection_loss‚0.5728531583396506 /loss/FBt1/detection_loss‚1.0245867256252539 /loss/FRt1/assignment_loss‚2.329081761258064 /loss/FRt1/detection_loss‚0.5372779953425562 1loss/SRqqt1/detection_loss‚0.9598945630328922 trainer/global_step‚7999 2loss/SRqqt1/assignment_loss‚0.6876888892645466 3loss/FBt2/assignment_loss‚0.0005278553941520449 /loss/FBt2/detection_loss‚0.5592180362933117 _runtime‚ 3273.08576764 2loss/SRqqt2/assignment_loss‚0.6080657393120059 3loss/FBt1/assignment_loss‚0.0032085538011409924 /loss/FRt2/detection_loss‚1.4173989803757003 %loss/total_loss‚9.639869311326937ˆ´.ŠÍeŽ‚(:Ú ªÆõÒÕØü¿ disk.md0.out‚107.75$ memory_percent‚99.22237396240234% proc.memory.availableMB‚ 127.40625( disk./.usagePercent‚75.7085609156502# disk.dm-0.out‚25104.33349609375 disk.dm-1.out‚ 3064.17578125" disk.nvme0c0n1.in‚ 106.19140625 disk.nvme1n1.in‚ 10.48828125$ disk.nvme1n1.out‚7914.0361328125) proc.memory.percent‚17.53997802734375 cpu‚26.45625267818961 network.sent‚ 1.0537793e+07% disk./.usageGB‚1162.3156814575195 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 106.19140625! disk.nvme1c1n1.in‚ 10.48828125 proc.memory.rssMB‚2873.75 network.recv‚936701 disk.rbd0.out‚0( disk.nvme0c0n1.out‚20486.56103515625 proc.cpu.threads‚17 disk.dm-0.in‚ 116.6796875& disk.nvme0n1.out‚20486.56103515625& disk.nvme1c1n1.out‚7914.0361328125 disk.md0.in‚0ÎÉî1›‚(: ªÆõÒçÚÀ gpu.0.gpu‚44.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚198.266' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.08866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óI¹72- _wandbruntime‚3291  _runtime‚3291º†¥–M(‘º{ 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5504531860351562 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5634092092514038 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.030186062678694725 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7089381217956543 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.496303915977478 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4475063681602478 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8470457792282104 +particle/specificity‚0.9081403017044067 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8316238522529602 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9980574250221252 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.025614136829972267 7validation_average_jet_accuracy‚0.34122252464294434 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_2_of_5‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10625344514846802 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9957804083824158 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 5 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.99006587266922 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 *jet/accuracy_1_of_1‚0.2936764657497406 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2095823734998703 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7956803441047668 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13773052394390106 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7217947840690613 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6783719062805176 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20085130631923676 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8258771300315857 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1316048949956894 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3756202757358551 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893944263458252 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7381993532180786 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9905549883842468 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6201401948928833 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6399307250976562 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11782453954219818 particle/accuracy_1_of_4‚1 )loss/val_total_loss‚9.350700511186195 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6918287873268127 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2777777910232544 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978741407394409 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16343152523040771 particle/accuracy_2_of_4‚1 jet/accuracy_1_of_5‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3757307827472687 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 )jet/accuracy_2_of_4‚0.786401093006134 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4616529047489166 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1502377688884735 jet/accuracy_5_of_5‚0 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1370152235031128 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12894518673419952 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.12975573539733887 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03129226714372635 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 epoch‚9 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8865830302238464 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9967795610427856 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8470457792282104 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19718381762504578 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9960682988166809 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +particle/sensitivity‚0.6247429251670837 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.07901572436094284 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 % _timestamp‚1.7845051377991092e+09 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1504388451576233 - Purity/2FRt*SRqqt1FBt/FRt_purity‚0.484375 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7210463285446167 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.08359488099813461 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11562500149011612 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8182844519615173 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5186704993247986 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6493695974349976 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9956815838813782 /particle/accuracy_4_of_4‚0.9403634667396545 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9750000238418579 trainer/global_step‚8009 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8688177466392517 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9648414850234985 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1447916179895401 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5666800737380981 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9951990842819214 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4381454586982727 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7385290861129761 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.09502904862165451 (particle/accuracy‚0.8472841382026672 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.631016194820404 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11605270951986313 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4418187439441681 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6274865865707397 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.030614012852311134 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3425925672054291 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5062435269355774 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5674388408660889 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14616481959819794 'particle/f_score‚0.6372318267822266 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23734891414642334 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5660382509231567 *jet/accuracy_1_of_3‚0.9065033197402954 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.039864785969257355 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5104166865348816 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0357142873108387 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9965552091598511 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6354126334190369 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2761356234550476 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7946133613586426 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5151082277297974 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.06656090915203094 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 *jet/accuracy_1_of_2‚0.7585633397102356 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13170844316482544 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1370152235031128 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0357142873108387 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.09686948359012604 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9951246380805969 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8242481350898743 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.05000000074505806 & Purity/2FRt0SRqqt1FBt/FBt_pÆ©‡surity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7438487410545349 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7544077038764954 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7536901235580444 'jet/accuracy_2_of_2‚0.1435536891222 particle/accuracy_1_of_3‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6849642992019653 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.014615441672503948 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006748279090970755 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15375760197639465 /particle/accuracy_2_of_2‚0.9998950958251953 particle/accuracy_3_of_3‚1 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998833417892456 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02701628766953945 *jet/accuracy_2_of_3‚0.5347207188606262 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.05000000074505806 particle/accuracy_3_of_5‚1 _step‚649 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7154086232185364 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4015151858329773 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.025614136829972267 particle/accuracy_3_of_4‚1 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5112724900245667 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6639368534088135 jet/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6431460380554199 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787886738777161 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6399307250976562 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_3‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.030535276979207993 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6851824522018433 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962481260299683 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7384366393089294 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7942981719970703 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 _runtime‚3291.863116876 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3925960659980774 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.610134482383728 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7441025972366333 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.012653161771595478 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4749999940395355 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9821428656578064 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6321806907653809 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.27272728085517883 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8465191721916199 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9196552038192749 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6563581824302673 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11748772859573364 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.014682539738714695 particle/accuracy_1_of_2‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9898113012313843 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7450591325759888 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.15161266922950745 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.25073352456092834 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 particle/accuracy_4_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.43525558710098267 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.030387531965970993 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17418307065963745 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 +jet/accuracy_3_of_3‚0.06842396408319473 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9309861063957214 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05‚ÁšÅ=’¿{ & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 5 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.99006587266922 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 *jet/accuracy_1_of_1‚0.2936764657497406 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2095823734998703 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7956803441047668 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13773052394390106 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7217947840690613 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6783719062805176 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20085130631923676 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8258771300315857 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1316048949956894 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3756202757358551 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9893944263458252 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7381993532180786 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9905549883842468 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6201401948928833 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6399307250976562 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.11782453954219818 particle/accuracy_1_of_4‚1 )loss/val_total_loss‚9.350700511186195 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6918287873268127 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2777777910232544 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978741407394409 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16343152523040771 particle/accuracy_2_of_4‚1 jet/accuracy_1_of_5‚1 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3757307827472687 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 )jet/accuracy_2_of_4‚0.786401093006134 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4616529047489166 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1502377688884735 jet/accuracy_5_of_5‚0 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1370152235031128 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12894518673419952 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.12975573539733887 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.03129226714372635 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 epoch‚9 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8865830302238464 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9967795610427856 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8470457792282104 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19718381762504578 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9960682988166809 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 +particle/sensitivity‚0.6247429251670837 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.07901572436094284 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 % _timestamp‚1.7845051377991092e+09 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1504388451576233 - Purity/2FRt*SRqqt1FBt/FRt_purity‚0.484375 particle/accuracy_1_of_5‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7210463285446167 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.08359488099813461 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.11562500149011612 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8182844519615173 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5186704993247986 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6493695974349976 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9956815838813782 /particle/accuracy_4_of_4‚0.9403634667396545 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9750000238418579 trainer/global_step‚8009 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8688177466392517 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.9648414850234985 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1447916179895401 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5666800737380981 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9951990842819214 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4381454586982727 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7385290861129761 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 +jet/accuracy_3_of_4‚0.09502904862165451 (particle/accuracy‚0.8472841382026672 8"Purity/0FRt1SRqqt*FBt/event_purity‚0.631016194820404 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.11605270951986313 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4418187439441681 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6274865865707397 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.030614012852311134 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3425925672054291 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5062435269355774 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5674388408660889 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14616481959819794 'particle/f_score‚0.6372318267822266 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.23734891414642334 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5660382509231567 *jet/accuracy_1_of_3‚0.9065033197402954 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.039864785969257355 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5104166865348816 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0357142873108387 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9965552091598511 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6354126334190369 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2761356234550476 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7946133613586426 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5151082277297974 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.06656090915203094 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 *jet/accuracy_1_of_2‚0.7585633397102356 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13170844316482544 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1370152235031128 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0357142873108387 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.09686948359012604 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9951246380805969 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8242481350898743 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.05000000074505806 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7438487410545349 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7544077038764954 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7536901235580444 'jet/accuracy_2_of_2‚0.1435536891222 particle/accuracy_1_of_3‚1 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6849642992019653 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.014615441672503948 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006748279090970755 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15375760197639465 /particle/accuracy_2_of_2‚0.9998950958251953 particle/accuracy_3_of_3‚1 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998833417892456 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02701628766953945 *jet/accuracy_2_of_3‚0.5347207188606262 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.05000000074505806 particle/accuracy_3_of_5‚1 _step‚649 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7154086232185364 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4015151858329773 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.025614136829972267 particle/accuracy_3_of_4‚1 jet/accuracy_2_of_5‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5112724900245667 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6639368534088135 jet/accuracy_3_of_5‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6431460380554199 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9787886738777161 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6399307250976562 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_3‚1 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.6875 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.030535276979207993 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6851824522018433 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962481260299683 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7384366393089294 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.7942981719970703 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 _runtime‚3291.863116876 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3925960659980774 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.610134482383728 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7441025972366333 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_5_of_5‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.012653161771595478 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4749999940395355 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9821428656578064 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6321806907653809 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.27272728085517883 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8465191721916199 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9196552038192749 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6563581824302673 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.11748772859573364 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,jet/accuracy_4_of_4‚0.014682539738714695 particle/accuracy_1_of_2‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9898113012313843 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7450591325759888 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.15161266922950745 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.25073352456092834 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 particle/accuracy_4_of_5‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.43525558710098267 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.030387531965970993 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17418307065963745 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 +jet/accuracy_3_of_3‚0.06842396408319473 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7142857313156128 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9309861063957214 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5504531860351562 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5634092092514038 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.030186062678694725 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7089381217956543 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.496303915977478 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4475063681602478 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8470457792282104 +particle/specificity‚0.9081403017044067 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8316238522529602 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9980574250221252 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.025614136829972267 7validation_average_jet_accuracy‚0.34122252464294434 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 particle/accuracy_2_of_5‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.10625344514846802 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9957804083824158‰þ?p“‚(:å ¹ÆõÒ©¡á¿% disk.nvme1c1n1.out‚8010.619140625# disk.nvme1n1.out‚8010.619140625 disk.md0.out‚ 108.2109375 disk.dm-0.in‚ 116.703125 disk.dm-1.in‚0 disk.rbd0.in‚0$ memory_percent‚98.82149696350098) proc.memory.percent‚17.54598617553711 network.sent‚ 1.0598965e+07% disk./.usageGB‚1163.4312934875488 disk.nvme1n1.in‚ 10.48828125 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 106.21484375% disk.nvme0n1.out‚21732.4697265625! proc.memory.rssMB‚ 2874.734375) disk./.usagePercent‚75.78122738886492! disk.nvme1c1n1.in‚ 10.48828125# disk.dm-0.out‚26384.97021484375 disk.nvme0n1.in‚ 106.21484375' proc.memory.availableMB‚ 193.0859375 cpu‚26.460376935055415 proc.cpu.threads‚17 network.recv‚940937 disk.md0.in‚0" disk.dm-1.out‚3128.18212890625' disk.nvme0c0n1.out‚21732.4697265625˜Þ˸”‚(:… ¹ÆõҌͶÀ gpu.0.gpu‚12.0 gpu.0.memory‚2+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.393' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.131 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ù×Rø2•- _wandbruntime‚3307  _runtime‚3307c”°`É–Ã ' lr-AdamW/pg1‚0.00010295396389197827 ' lr-AdamW/pg2‚0.00010295396389197827 trainer/global_step‚8049 % _timestamp‚1.7845051532507641e+09 _runtime‚3307.313883873 _step‚650–vVÈÎ—È ' lr-AdamW/pg2‚0.00010295396389197827 trainer/global_step‚8049 % _timestamp‚1.7845051532507641e+09 _runtime‚3307.313883873 _step‚650 ' lr-AdamW/pg1‚0.00010295396389197827Š[#†2˜- _wandbruntime‚3307  _runtime‚3307½˜…NŽ%™ÒO =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚316203850728448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5650845066130 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5668452312064 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5834443969024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61312719 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚316203850728448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5668540334080 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81656952 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚306514303669232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5668540334080 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚302451202909696 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚312165045547528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _runtime‚3307.314590084 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚296616783844352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142969671 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚316203745667072 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚310535310394368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20679751 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56157235 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81656952 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚310535293355008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚310535310394368 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 % _timestamp‚1.7845051532515678e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20679984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61312719 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚302451416615424 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚8049 N@DeviceStatsMon,„ŒeJitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316203745667072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35477484 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35477487ßÔ Ý'š×O \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚310535293355008 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5668452312064 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚8049 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316203745667072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35477484 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35477487 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚316203850728448 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5650845066130 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5668452312064 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5834443969024 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61312719 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚316203850728448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5668540334080 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81656952 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚306514303669232 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5668540334080 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚302451202909696 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚312165045547528 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _runtime‚3307.314590084 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚296616783844352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 142969671 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚233 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚316203745667072 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚310535310394368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20679751 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56157235 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81656952 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚310535293355008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚236 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚310535310394368 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 % _timestamp‚1.7845051532515678e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20679984 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61312719 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚302451416615424 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚296652479001600 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56163898 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142986904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142986904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5669130421760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81656707 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 % _timestamp‚1.7845051535117354e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5669218450944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚ 3307.57523594 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35481777 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 142976745 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61320040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56164144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚312202593566732 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚316241895212544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚316241895212544 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚306551175736992 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚310572676761600 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚306551158697632 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81666864 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56163898 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 142986904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 142986904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5669130421760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81656707 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 % _timestamp‚1.7845051535117354e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5669218450944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _runtime‚ 3307.57523594 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35481777 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 142976745 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61320040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56164144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚312202593566732 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚316241895212544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚316241895212544 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚306551175736992 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚310572676761600 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚306551158697632 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚246 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81666864 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚296652479001600 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61678769 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5702478075904 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚308357890017872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82143561 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56494897 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚298401541706752 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5702390053888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚308357907057232 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20802913 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5684588393064 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5684674541538 _step‚655 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚8099 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚318106001770496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚304270907601408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚314042581598770 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5702390053888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7845051723144958e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56495160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚318106001770496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0÷g¨Ý'¨×O N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82143561 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56494897 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚298401541706752 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5702390053888 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚308357907057232 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20802913 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5684588393064 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5684674541538 _step‚655 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚8099 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚318106001770496 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚304270907601408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚314042581598770 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5702390053888 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7845051723144958e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56495160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚318106001770496 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚298401516803072 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚312403628755968 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82153704 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚314042478410936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚312403611716608 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143822328 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5869573308928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 _runtime‚3326.377526491 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 143832473 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35691984 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20803173 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61678769 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5702478075904 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚308357890017872¡e¿2©- _wandbruntime‚3326  _runtime‚3326y%á&ªÛM F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61686088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5685350493094 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82163616 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚308394762085632 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5870274991104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 143839547 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5703068163584 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61686088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143849706 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚298437211960320 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20805344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143849706 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5703156192768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20805593 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5870086196224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61686090 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56501618 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚312440978083840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚312440995123200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304307511855104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚312440978083840 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56501870 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚314080129618086 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚318144046247424 trainer/global_step‚8099 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚312440995123200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82153459 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5703068163584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35696277 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5703156192768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5685264344508 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 $ _timestamp‚1.784505172577086e+09ƒŒ¶‰J«àM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚304307298156544 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J=DeviceStatsMonitor.on_train_batch_end/ac&»‰œ#tive.large_pool.freed‚61686088 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5685350493094 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82163616 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚308394762085632 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5870274991104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 143839547 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5703068163584 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61686088 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143849706 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚298437211960320 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20805344 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143849706 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5703156192768 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20805593 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5870086196224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61686090 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56501618 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚312440978083840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚312440995123200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304307511855104 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚312440978083840 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56501870 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚314080129618086 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚318144046247424 trainer/global_step‚8099 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚312440995123200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚82153459 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5703068163584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35696277 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5703156192768 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5685264344508 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 $ _timestamp‚1.784505172577086e+09 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320008257873920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35906484 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82640313 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35906487 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5718504016946 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚314271947117568 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 _runtime‚3345.859597318 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 % _timestamp‚1.7845051917965522e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82650456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62044819 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚314271930078208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56832765 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5904533044224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5718417868472 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144685130 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82650456 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚314271947117568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62044817 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5736327795712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚300186274665472 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚320008362935296 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144695275 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚310201510445232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚306090832613376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚314271930078208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20926022 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚306090618907648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360÷¶È”Õ!¸×O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320008257873920 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35906484 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚82640313 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35906487 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5718504016946 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚314271947117568 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 _runtime‚3345.859597318 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 % _timestamp‚1.7845051917965522e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82650456 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62044819 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚314271930078208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56832765 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5904533044224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5718417868472 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144685130 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚82650456 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚314271947117568 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62044817 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5736327795712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚300186274665472 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚320008362935296 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144695275 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚310201510445232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚306090832613376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚314271930078208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20926022 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚306090618907648 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62044819 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5736415817728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚315920014462178 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82640313 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚8149 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freedÞmþ‚320008257873920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5736327795712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56832506 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5736415817728 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚320008362935296 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚300186299569152 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0“œx½‡2¹- _wandbruntime‚3346  _runtime‚3346@â&ºÜM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56839540 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _runtime‚3346.125383026 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62052140 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62052140 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5719179968502 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35910774 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚315957459293548 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 % _timestamp‚1.7845051920620494e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62052138 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62052138 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚314309313484800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82660368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚306127012471296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚660 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5905042648576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 trainer/global_step‚8149 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚300221969822720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚315957562481494 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35910777 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144702349 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚314309313484800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20928505 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56839279 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144712508 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712¶Äþç&»áM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62052140 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62052140 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5719179968502 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35910774 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚315957459293548 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 % _timestamp‚1.7845051920620494e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62052138 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62052138 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚314309313484800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚82660368 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚306127012471296 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚660 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5905042648576 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 trainer/global_step‚8149 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚300221969822720 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚315957562481494 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35910777 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 144702349 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚314309313484800 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20928505 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56839279 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144712508 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56839540 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _runtime‚3346.125383026”áX2¼- _wandbruntime‚3346  _runtime‚33466¡˜{ ½ƒ _step‚661 2loss/SRqqt1/assignment_loss‚0.7830297668050561 1loss/SRqqt1/detection_loss‚1.0959150376212266 1loss/SRqqt2/detection_loss‚0.7023397717562642 0loss/FRt1/assignment_loss‚2.3173837655685783 2loss/SRqqt2/assignment_loss‚0.6087001932155494 /loss/FRt2/detection_loss‚1.5510793595550287 /loss/FBt2/detection_loss‚0.5285830876984012 &loss/total_loss‚10.309946461335553 0loss/FRt2/assignment_loss‚0.8807267954475001 4loss/FBt1/assignment_loss‚0.00030849549163036727 .loss/FBt1/detection_loss‚1.236519779311493 trainer/global_step‚8149 2loss/FBt2/assignment_loss‚0.024769103351416532 -loss/FRt1/detection_loss‚0.58059130551341 epoch‚10 $ _timestamp‚1.784505192063022e+09 _runtime‚3346.125865237º«Ó¾ˆ epoch‚10 $ _timestamp‚1.784505192063022e+09 _runtime‚3346.125865237 _step‚661 2loss/SRqqt1/assignment_loss‚0.7830297668050561 1loss/SRqqt1/detection_loss‚1.0959150376212266 1loss/SRqqt2/detection_loss‚0.7023397717562642 0loss/FRt1/assignment_loss‚2.3173837655685783 2loss/SRqqt2/assignment_loss‚0.6087001932155494 /loss/FRt2/detection_loss‚1.5510793595550287 /loss/FBt2/detection_loss‚0.5285830876984012 &loss/total_loss‚10.309946461335553 0loss/FRt2/assignment_loss‚0.8807267954475001 4loss/FBt1/assignment_loss‚0.00030849549163036727 .loss/FBt1/detection_loss‚1.236519779311493 trainer/global_step‚8149 2loss/FBt2/assignment_loss‚0.024769103351416532 -loss/FRt1/detection_loss‚0.58059130551341•YÀ!Üq¿‚(:æ õÆõÒ¿Äì¿% disk.nvme1c1n1.out‚10403.87109375 disk.nvme1n1.in‚ 10.48828125 disk.md0.out‚ 109.6015625 disk.dm-0.in‚ 116.7265625 disk.dm-1.in‚0( proc.memory.availableMB‚ 192.88671875 disk.nvme0n1.in‚ 106.23828125# disk.nvme1n1.out‚10403.87109375 disk.md0.in‚0! disk.dm-1.out‚3192.6044921875" disk.nvme0c0n1.in‚ 106.23828125# memory_percent‚98.8227128982544! proc.memory.rssMB‚ 2874.734375) proc.memory.percent‚17.54598617553711 network.sent‚ 1.0785338e+07# disk.dm-0.out‚29765.14404296875 disk.rbd0.out‚0 cpu‚26.458486779835766 network.recv‚956975$ disk./.usageGB‚1163.779640197754! disk.nvme1c1n1.in‚ 10.48828125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚22786.80908203125 proc.cpu.threads‚17) disk./.usagePercent‚75.80391729019729& disk.nvme0n1.out‚22786.80908203125—ïÒ+‘À‚(:† õÆõÒ——µÀ gpu.0.gpu‚75.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚201.498' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚67.166 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üsÕ»2Á- _wandbruntime‚3365  _runtime‚3365¯fÉÂà % _timestamp‚1.7845052115839083e+09 _runtime‚3365.647055926 _step‚662 ' lr-AdamW/pg1‚0.00010258162312630617 ' lr-AdamW/pg2‚0.00010258162312630617 trainer/global_step‚8199™ÌýìÎÃÈ ' lr-AdamW/pg1‚0.00010258162312630617 ' lr-AdamW/pg2‚0.00010258162312630617 trainer/global_step‚8199 % _timestamp‚1.7845052115839083e+09 _runtime‚3365.647055926 _step‚662–`ê~d2Ä- _wandbruntime‚3365  _runtime‚3365Ñà`|UÅÒO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36120984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5939456794624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83147208 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚321910619038720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36120987 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21049619 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57170603 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚3365.647971578 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 145558077 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚301971032527872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚316140265479168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 % _timestamp‚1.7845052115848637e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚321910513977344 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5752247343880 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.fr:H–ƒ eed‚62410867 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5770353559552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5770353559552 _step‚663 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21049869 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚312045113833232 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5770265537536 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83137065 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚8199 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚321910513977344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62410869 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83147208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5752333492354 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚316140265479168 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚316140248439808 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104¶¯wÝ'Æ×O SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62410867 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5770353559552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5770353559552 _step‚663 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21049869 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚312045113833232 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5770265537536 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83137065 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 trainer/global_step‚8199 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚321910513977344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62410869 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83147208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5752333492354 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚316140265479168 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚316140248439808 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36120984 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5939456794624 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83147208 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚321910619038720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36120987 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21049619 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57170603 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚3365.647971578 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 145558077 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚301971032527872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚316140265479168 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 % _timestamp‚1.7845052115848637e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚321910513977344 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5752247343880—ç.2Ç- _wandbruntime‚3365  _runtime‚3365jŸšäá&ÈÛM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145575310 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62418188 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚316177631846400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5771031676416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83157120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚312081968861632 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5770943647232 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚317834892156956 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145565151 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83157120 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5771031676416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚316177614807040 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62418190 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57177382 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83146963 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 $ _timestamp‚1.784505211848238e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21052108 _step‚664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5770943647232 _runtime‚3365.911598988 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62418190 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚302006727685120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36125274 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚321948663522816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚317834995344902 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21052358 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚302006752588800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145575310 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62418188 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482WÄT)æ&ÉàM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚321948663522816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚317834995344902 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21052358 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚302006752588800 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 145575310 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62418188 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 145575310 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62418188 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚316177631846400 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5771031676416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83157120 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚312081968861632 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5770943647232 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚317834892156956 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 145565151 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83157120 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5771031676416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚316177614807040 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62418190 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57177382 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83146963 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 $ _timestamp‚1.784505211848238e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21052108 _step‚664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5770943647232 _runtime‚3365.911598988 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62418190 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚302006727685120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36125274 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0˜–i«P2Ê- _wandbruntime‚3365  _runtime‚3365×õˆDËþ 0loss/FRt1/assignment_loss‚2.0212870149978444 /loss/FBtR¦XqÀ2/detection_loss‚0.5299773636017588 trainer/global_step‚8199 2loss/SRqqt1/assignment_loss‚0.6760692653980406 0loss/FBt1/assignment_loss‚0.0144639726525232 .loss/FRt2/detection_loss‚1.498312919821945 0loss/SRqqt2/detection_loss‚0.635090503433693 .loss/FBt1/detection_loss‚0.647546528528134 % _timestamp‚1.7845052118496504e+09 0loss/FRt2/assignment_loss‚0.7327016403664758 2loss/FBt2/assignment_loss‚0.009335711613638267 1loss/SRqqt1/detection_loss‚1.0725420115270863 epoch‚10 _runtime‚ 3365.91252999 2loss/SRqqt2/assignment_loss‚0.6326208205431254 /loss/FRt1/detection_loss‚0.5794817230895101 %loss/total_loss‚9.049429475573774 _step‚665] ‹O ̃ 0loss/FBt1/assignment_loss‚0.0144639726525232 .loss/FRt2/detection_loss‚1.498312919821945 0loss/SRqqt2/detection_loss‚0.635090503433693 .loss/FBt1/detection_loss‚0.647546528528134 % _timestamp‚1.7845052118496504e+09 0loss/FRt2/assignment_loss‚0.7327016403664758 2loss/FBt2/assignment_loss‚0.009335711613638267 1loss/SRqqt1/detection_loss‚1.0725420115270863 epoch‚10 _runtime‚ 3365.91252999 2loss/SRqqt2/assignment_loss‚0.6326208205431254 /loss/FRt1/detection_loss‚0.5794817230895101 %loss/total_loss‚9.049429475573774 _step‚665 0loss/FRt1/assignment_loss‚2.0212870149978444 /loss/FBt2/detection_loss‚0.5299773636017588 trainer/global_step‚8199 2loss/SRqqt1/assignment_loss‚0.6760692653980406™ÂlbŠoÍ‚(:ä „ÇõÒ˜¶½¿) disk./.usagePercent‚75.97572886969053! disk.dm-1.out‚3194.7763671875 disk.nvme1c1n1.in‚ 10.5859375 disk.nvme1n1.in‚ 10.5859375 disk.md0.out‚ 112.40234375! proc.memory.rssMB‚ 2874.734375% disk./.usageGB‚1166.4173774719238 disk.rbd0.out‚0 disk.nvme0n1.in‚ 106.6640625& disk.nvme0n1.out‚23655.38818359375 disk.md0.in‚0$ memory_percent‚98.82311820983887' proc.memory.availableMB‚ 192.8203125) proc.memory.percent‚17.54598617553711 network.sent‚ 1.0848733e+07 disk.dm-0.in‚117.25" disk.dm-0.out‚30799.3544921875 disk.dm-1.in‚0( disk.nvme0c0n1.out‚23655.38818359375( disk.nvme1c1n1.out‚10577.36474609375 network.recv‚961274 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 106.6640625& disk.nvme1n1.out‚10577.36474609375 cpu‚26.462291113475807 proc.cpu.threads‚17ZÁåӛ΂(: „ÇõÒèòÀ gpu.0.gpu‚48.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚112.24' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.413333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04ÑLÂ2Ï- _wandbruntime‚3385  _runtime‚3385ÊïÝDÉÐà _step‚666 ' lr-AdamW/pg1‚0.00010245601671113134 ' lr-AdamW/pg2‚0.00010245601671113134 trainer/global_step‚8249 % _timestamp‚1.7845052311248481e+09 _runtime‚3385.188039727ÒÝ܃ÎÑÈ trainer/global_step‚8249 % _timestamp‚1.7845052311248481e+09 _runtime‚3385.188039727 _step‚666 ' lr-AdamW/pg1‚0.00010245601671113134 ' lr-AdamW/pg2‚0.00010245601671113134šËù¿$2Ò- _wandbruntime‚3385  _runtime‚3385´ãd…Ø'ÓÒO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5804291301376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36335484 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚319674777001160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚323812875142144 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57509721 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚319674880188994 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62776919 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83643960 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83643960 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚313888717221232 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5786162967762 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21173963 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚323812770080768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5804203279360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83633817 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21174234 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚303755790390272 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5974231258112 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚313888700181872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚3385.190063871 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 146410734 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146420879 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62776919 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57509447 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5786076819288 trainer/global_step‚8249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚318008566801408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚309730241645568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚309730021648384 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219997184 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323812770080768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36335487 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚318008583840768 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5804291301376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 195093504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚318008566801408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚309730241645568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚309730021648384 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚271 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 219997184 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323812770080768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36335487 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚318008583840768 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5804291301376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 195093504 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚274 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚313888717221232 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5786162967762 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21173963 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚323812770080768 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5804203279360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83633817 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21174234 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚303755790390272 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5974231258112 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚313888700181872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _runtime‚3385.190063871 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 146410734 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146420879 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62776919 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57509447 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5786076819288 trainer/global_step‚8249 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 J2Õ- _wandbruntime‚3385  _runtime‚3385 d¦w÷"ÖÜM BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36339774 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚323850919626240 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚323850814557696 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83643715 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚319712325020364 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5804881389056 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5786838919318 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚313925572249632 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62784240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57516163 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 _step‚668 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83653872 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚318045950208000 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚318045933168640 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5804969418240 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚309766426297856 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5786752770732 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83643715 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21176389 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚62784240 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 146427953 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146427953 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚303791485547520 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36339777 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5974940750336 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146438112 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚313925589288992 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _runtime‚3385.756847934 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5804881389056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5975116962304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚303791510451200 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62784238 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚323850919626240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/—gl4ëinactive_split_bytes.all.allocated‚309766627413504 trainer/global_step‚8249 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146438112 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚313925589288992 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _runtime‚3385.756847934 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5804881389056 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5975116962304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚303791510451200 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62784238 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚323850919626240 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚309766627413504 trainer/global_step‚8249 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36339774 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚323850919626240 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚323850814557696 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83643715 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚319712325020364 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5804881389056 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5786838919318 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚313925572249632 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62784240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57516163 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 _step‚668 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚83653872 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0œ²Ì™@2Ø- _wandbruntime‚3385  _runtime‚3385ÀE Ùƒ 1loss/SRqqt1/detection_loss‚1.1116550496091633 .loss/FBt2/detection_loss‚0.676072096908465 /loss/FRt1/assignment_loss‚2.144284847248123 2loss/SRqqt1/assignment_loss‚0.6583451615578507 3loss/FBt2/assignment_loss‚0.0025035162777645326 1loss/SRqqt2/detection_loss‚0.6300020991830946 /loss/FBt1/detection_loss‚1.4778162849446137 trainer/global_step‚8249 % _timestamp‚1.7845052316950357e+09 _runtime‚3385.758392317 epoch‚10 1loss/FBt1/assignment_loss‚0.10577917117536041 /loss/FRt1/detection_loss‚0.5521731306819483 &loss/total_loss‚10.100683023629461 _step‚669 0loss/FRt2/assignment_loss‚0.7622887249435147 2loss/SRqqt2/assignment_loss‚0.5597616457043667 /loss/FRt2/detection_loss‚1.4200012953951955?MyÚˆ 3loss/FBt2/assignment_loss‚0.0025035162777645326 1loss/SRqqt2/detection_loss‚0.6300020991830946 /loss/FBt1/detection_loss‚1.4778162849446137 trainer/global_step‚8249 % _timestamp‚1.7845052316950357e+09 _runtime‚3385.758392317 epoch‚10 1loss/FBt1/assignment_loss‚0.10577917117536041 /loss/FRt1/detection_loss‚0.5521731306819483 &loss/total_loss‚10.100683023629461 _step‚669 0loss/FRt2/assignment_loss‚0.7622887249435147 2loss/SRqqt2/assignment_loss‚0.5597616457043667 /loss/FRt2/detection_loss‚1.4200012953951955 1loss/SRqqt1/detection_loss‚1.1116550496091633 .loss/FBt2/detection_loss‚0.676072096908465 /loss/FRt1/assignment_loss‚2.144284847248123 2loss/SRqqt1/assignment_loss‚0.6583451615578507à{"nÛ‚(:ã “ÇõÒõÆô¿ proc.cpu.threads‚17 network.recv‚965256 disk.nvme1c1n1.in‚ 10.9296875& disk.nvme1n1.out‚10579.76513671875 disk.md0.out‚ 113.6796875 disk.dm-0.in‚ 123.84375' disk.nvme0c0n1.out‚26746.9970703125$ memory_percent‚98.82340431213379% disk./.usageGB‚1166.6014366149902 disk.nvme0n1.in‚ 112.9140625 disk.rbd0.out‚0! proc.memory.rssMB‚ 2874.734375% disk.nvme0n1.out‚26746.9970703125 disk.nvme1n1.in‚ 10.9296875 disk.dm-1.out‚3195.251953125! disk.nvme0c0n1.in‚ 112.9140625' proc.memory.availableMB‚ 192.7734375) proc.memory.percent‚17.54598617553711 cpu‚26.465020718475085 network.sent‚ 1.0911196e+07( disk./.usagePercent‚75.9877177407582( disk.nvme1c1n1.out‚10579.76513671875 disk.md0.in‚0# disk.dm-0.out‚33890.25732421875 disk.dm-1.in‚0 disk.rbd0.in‚0†h3éÜ‚(:… “ÇõÒÓÞÊÀ gpu.0.gpu‚47.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚205.137' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚68.379 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð#¹mÝ‚(:â ¢ÇõÒßÏú¿( proc.memory.availableMB‚ 128.11328125 cpu‚26.468244427891847 network.sent‚ 1.0974339e+07% disk./.usageGB‚1168.3657150268555 disk.dm-1.in‚0 disk.nvme1n1.in‚ 12.11328125% disk.nvme1n1.out‚10582.3876953125" disk.dm-0.out‚34047.6005859375 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 113.57421875$ disk.nvme0n1.out‚26902.798828125 disk.md0.in‚0# memory_percent‚99.2180585861206! proc.memory.rssMB‚ 2874.734375) proc.memory.percent‚17.54598617553711 proc.cpu.threads‚17 network.recv‚969526 disk.dm-1.out‚3195.322265625 disk.rbd0.in‚0! disk.nvme1c1n1.in‚ 12.11328125) disk./.usagePercent‚76.10263572882954& disk.nvme0c0n1.out‚26902.798828125 disk.nvme0n1.in‚ 113.57421875' disk.nvme1c1n1.out‚10582.3876953125 disk.md0.out‚ 114.1484375 disk.dm-0.in‚125.6875Ru`>œÞ‚(:‘ ¢ÇõÒ‡ÕªÀ gpu.0.gpu‚47.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚169.819' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚56.606333333333325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ò¦N×2ß- _wandbruntime‚3405  _runtime‚34054+aÈà trainer/global_step‚8299 $ _timestamp‚1.784505251030767e+09 _runtime‚3405.093876322 _step‚670 ' lr-AdamW/pg1‚0.00010232966722576691 ' lr-AdamW/pg2‚0.00010232966722576691a"åSÍáÇ $ _timestamp‚1.784505251030767e+09 _runtime‚3405.093876322 _step‚670 ' lr-AdamW/pg1‚0.00010232966722576691 ' lr-AdamW/pg2‚0.00010232966722576691 trainer/global_step‚8299ž¶0³Û2â- _wandbruntime‚3405  _runtime‚3405Ð8#PØ'ãÒO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63142967 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84130569 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6009214584832 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚325715131245568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84130569 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5838141021184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚315732303569872 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚305540573156352 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚311549762837504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84140712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚321552209864568 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚325715026184192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚3405.094704893 _step‚671 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 147273536 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚319876902202368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚319876885163008 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845052510316515e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚319876902202368 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5838229043200 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84140712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚305540548252672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚315732320609232‚wÓ¤ä×O WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚305540573156352 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚311549762837504 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84140712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚321552209864568 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚325715026184192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚3405.094704893 _step‚671 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 147273536 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocatÅÛÅ·9 ed‚319876902202368 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚319876885163008 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845052510316515e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚319876902202368 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5838229043200 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84140712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚305540548252672 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚315732320609232 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63142967 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84130569 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6009214584832 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚325715131245568 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚264 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84130569 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5838141021184 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚315732303569872ŸMñÙÁ2å- _wandbruntime‚3405  _runtime‚3405OV— â&æÜM ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚315769175637632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚321589861071718 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚315769192676992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚311586369716736 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84140467 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57854721 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚8299 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57854979 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989184 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84140467 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36554274 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5838819130880 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚325753175729664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6010101403136 _step‚672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63150290 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147290755 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84150624 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147300914 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36554277 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚305576243409920 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 _runtime‚3405.356740048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚319914251530240 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚325753070661120 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚319914268569600 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36554274 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5838819130880 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚325753175729664 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6010101403136 _step‚672 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63150290 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 147290755 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84150624 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 147300914 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36554277 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚305576243409920 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 _runtime‚3405.356740048 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚319914251530240 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚325753070661120 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚319914268569600 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84140467 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57854721 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚8299 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚57854979 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989184 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 Ï•¿2è- _wandbruntime‚3405  _runtime‚3405VŽâ_éÿ 0loss/FRt1/assignment_loss‚2.2049278387379263 2loss/FBt2/assignment_loss‚0.004738604637660966 .loss/FRt1/detection_loss‚0.548085241369794 1loss/SRqqt2/detection_loss‚0.6900971219320874 $loss/total_loss‚9.33725109498837 epoch‚10 _runtime‚3405.357213438 1loss/SRqqt1/assignment_loss‚0.655807504707099 2loss/SRqqt2/assignment_loss‚0.6761798335792676 1loss/SRqqt1/detection_loss‚0.9267993458763581 trainer/global_step‚8299 0loss/FRt2/assignment_loss‚0.9450304888159156 1loss/FBt1/assignment_loss‚0.08432351246123236 /loss/FRt2/detection_loss‚1.3984326284593973 _step‚673 /loss/FBt1/detection_loss‚0.7846659046375531 .loss/FBt2/detection_loss‚0.418163069774077 % _timestamp‚1.7845052512943964e+09©|ª ê„ /loss/FBt1/detection_loss‚0.7846659046375531 .loss/FBt2/detection_loss‚0.418163069774077 % _timestamp‚1.7845052512943964e+09 0loss/FRt1/assignment_loss‚2.2049278387379263 2loss/FBt2/assignment_loss‚0.004738604637660966 .loss/FRt1/detection_loss‚0.548085241369794 1loss/SRqqt2/detection_loss‚0.6900971219320874 $loss/total_loss‚9.33725109498837 epoch‚10 _runtime‚3405.357213438 1loss/SRqqt1/assignment_loss‚0.655807504707099 2loss/SRqqt2/assignment_loss‚0.6761798335792676 1loss/SRqqt1/detection_loss‚0.9267993458763581 trainer/global_step‚8299 0loss/FRt2/assignment_loss‚0.9450304888159156 1loss/FBt1/assignment_loss‚0.08432351246123236 /loss/FRt2/detection_loss‚1.3984326284593973 _step‚673¡„úËGtë‚(:é ±ÇõÒý¹¾¿) disk./.usagePercent‚76.10238253381155& disk.nvme0n1.out‚27929.07763671875! disk.nvme1c1n1.in‚ 12.11328125 disk.md0.out‚ 114.5546875 disk.dm-1.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 128.08203125 proc.cpu.threads‚17 network.sent‚ 1.0977261e+07" disk.nvme0c0n1.in‚ 113.57421875( disk.nvme0c0n1.out‚27929.07763671875 disk.nvme0n1.in‚ 113.57421875' disk.nvme1c1n1.out‚10583.4365234375% disk.nvme1n1.out‚10583.4365234375! proc.memory.rssMB‚ 2874.734375) proc.memory.percent‚17.54598617553711 disk.nvme1n1.in‚ 12.11328125 disk.md0.in‚0 disk.dm-0.in‚125.6875" disk.dm-0.out‚35073.6474609375" disk.dm-1.out‚3195.73291015625 disk.rbd0.out‚0 network.recv‚972532% disk./.usageGB‚1168.3618278503418$ memory_percent‚99.21824932098389 cpu‚26.471593924678213Ó«Ù‘ì‚(:† ±ÇõÒ‹—šÀ gpu.0.gpu‚85.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚197.145' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚65.715 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬ÿ̶2í- _wandbruntime‚3425  _runtime‚34259\'8Èî ' lr-AdamW/pg1‚0.00010220257669395635 ' lr-AdamW/pg2‚0.00010220257669395635 trainer/global_step‚8349 $ _timestamp‚1.784505271060918e+09 _runtime‚3425.124030023 _step‚674¸­òÍïÇ trainer/global_step‚8349 $ _timestamp‚1.784505271060918e+09 _runtime‚3425.124030023 _step‚674 ' lr-AdamW/pg1‚0.00010220257669395635 ' lr-AdamW/pg2‚0.00010220257669395635¢S×?P2ð- _wandbruntime‚3425  _runtime‚3425zçÇ+ñÒO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 _step‚675 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚321745203524608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 trainer/global_step‚8349 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36764487 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6044387208192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚321745220563968 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5853821918578 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84637464 _runtime‚3425.124943484 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21422309 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148136338 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚307325306115072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36764484 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚323429642727976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63509017 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5872078763008 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63509017 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚327617282287616 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148146483 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58186793 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚313369718226944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63509019 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5853735770104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?Devic1 º­eStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚317575923997232 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚327617282287616 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚327617387348992 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84627321 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21422560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚307325331018752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚321745203524608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚317575906957872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚313369504521216 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5872166785024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63509019 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚321745220563968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚327617387348992 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58186793 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚313369718226944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63509019 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5853735770104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚317575923997232 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚327617282287616 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚327617387348992 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚84627321 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21422560 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚307325331018752 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚321745203524608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚317575906957872 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚313369504521216 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5872166785024 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63509019 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚321745220563968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚327617387348992 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84637464 _runtime‚3425.124943484 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21422309 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148136338 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚307325306115072 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36764484 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚323429642727976 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63509017 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5872078763008 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚63509017 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚327617282287616 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148146483 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63516340 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5872844901888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚321782586931200 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84637219 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 148153557 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84637219 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6045084743168 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚323467293935126 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚323467190747180 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚307361001272320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58193489 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚321782586931200 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 _step‚676 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚327655326764544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84647376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 _runtime‚3425.392676946 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58193767 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚317612796064992 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5854411721548 trainer/global_step‚8349 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63516340 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚317612779025632 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚307361026176000 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63516338 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 % _timestamp‚1.7845052713293743e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63516338 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84647376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36768777 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148163716 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚327655431833088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚313405897220608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36768774 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚321782569891840 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚321782569891840 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 148153557 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5854497870134Õ‹n? õáM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5854411721548 trainer/global_step‚8349 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚63516340 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚317612779025632 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚307361026176000 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63516338 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 % _timestamp‚1.7845052713293743e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63516338 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84647376 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36768777 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148163716 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚327655431833088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚313405897220608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36768774 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚278 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚321782569891840 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚321782569891840 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 148153557 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5854497870134 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚327655431833088 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5872756872704 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6044895948288 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21424990 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148163716 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚63516340 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5872844901888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚321782586931200 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84637219 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 148153557 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84637219 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6045084743168 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚323467293935126 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚323467190747180 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚307361001272320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58193489 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚321782586931200 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 _step‚676 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚327655326764544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚84647376 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 _runtime‚3425.392676946 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58193767 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚317612796064992¤»9 Å2ö- _wandbruntime‚3425  _runtime‚3425ÈSµ ÷„ 2loss/FBt1/assignment_loss‚0.024034474074622714 /loss/FRt2/detection_loss‚1.4638661699311342 &loss/total_loss‚10.025458270485949 /loss/FRt1/assignment_loss‚2.228941218354093 2loss/SRqqt1/assignment_loss‚0.6120611121850864 2loss/FBt2/assignment_loss‚0.001079130542422134 /loss/FRt1/detection_loss‚0.6117842826269421 epoch‚10 2loss/SRqqt2/assignment_loss‚0.5986406497921926 1loss/SRqqt2/detection_loss‚0.7161942767410007 /loss/FBt2/detection_loss‚0.5656272497144528 trainer/global_step‚8349 1loss/SRqqt1/detection_loss‚1.1250220645902058 /loss/FBt1/detection_loss‚1.1494700581773514 % _timestamp‚1.7845052713311596e+09 _runtime‚3425.394232759 _step‚677 0loss/FRt2/assignment_loss‚0.9287375837564469‰š°Âø‰ 1loss/SRqqt1/detection_loss‚1.1250220645902058 /loss/FBt1/detection_loss‚1.1494700581773514 % _timestamp‚1.7845052713311596e+09 _runtime‚3425.394232759 _step‚677 0loss/FRt2/assignment_loss‚0.9287375837564469 2loss/FBt1/assignment_loss‚0.024034474074622714 /loss/FRt2/detection_loss‚1.4638661699311342 &loss/total_loss‚10.025458270485949 /loss/FRt1/assignment_loss‚2.228941218354093 2loss/SRqqt1/assignment_loss‚0.6120611121850864 2loss/FBt2/assignment_loss‚0.001079130542422134 /loss/FRt1/detection_loss‚0.6117842826269421 epoch‚10 2loss/SRqqt2/assignment_loss‚0.5986406497921926 1loss/SRqqt2/detection_loss‚0.7161942767410007 /loss/FBt2/detection_loss‚0.5656272497144528 trainer/global_step‚8349¥)‘žšù‚(: ÀÇõÒÜ“À gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.67' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.223333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Gv›$rú‚(:ç ÀÇõÒýÍœÄ! proc.memory.rssMB‚ 2874.734375 network.sent‚ 1.1040802e+07 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚27941.2822265625 disk.nvme1n1.in‚ 12.11328125 cpu‚26.47446809176653) disk./.usagePercent‚76.10232538478884 disk.dm-0.in‚125.6875" disk.nvme0c0n1.in‚ 113.57421875! disk.nvme1c1n1.in‚ 12.11328125( disk.nvme1c1n1.out‚10728.81494140625 disk.md0.out‚ 114.8828125( proc.memory.availableMB‚ 127.76953125 proc.cpu.threads‚17 network.recv‚977176" disk.dm-0.out‚35230.3955078125" disk.dm-1.out‚3195.85009765625' disk.nvme0c0n1.out‚27941.2822265625& disk.nvme1n1.out‚10728.81494140625 disk.md0.in‚0# memory_percent‚99.2201566696167) proc.memory.percent‚17.54598617553711% disk./.usageGB‚1168.3609504699707 disk.nvme0n1.in‚ 113.57421875yo&2û- _wandbruntime‚3445  _runtime‚3445 ¦ Éüà _step‚678 ' lr-AdamW/pg1‚0.00010207474715131248 ' lr-AdamW/pg2‚0.00010207474715131248 trainer/global_step‚8399 % _timestamp‚1.7845052912866938e+09 _runtime‚3445.349880193Š?ÎÎýÈ % _timestamp‚1.7845052912866938e+09 _runtime‚3445.349880193 _step‚678 ' lr-AdamW/pg1‚0.00010207474715131248 ' lr-AdamW/pg2‚0.00010207474715131248 trainer/global_step‚8399¦åöÄ2þ- _wandbruntime‚3445  _runtime‚3445¢ ¦bØ'ÿÒO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚325307075591384 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85124073 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149009285 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚319419510345872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329519538391040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5906104526848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63875067 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36978987 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚323613538925568 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚323613521886208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63875069 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199025664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58525591 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63875069 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6079200666112 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 % _timestamp‚1.7845052912875915e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85134216 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148999140 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85124073 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36978984 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚319419527385232 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21546607 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5906104526848 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329519538391040 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21546863 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0p¤tgÝ'€×O \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58525591 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚63875069 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6079200666112 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 % _timestamp‚1.7845052912875915e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85134216 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148999140 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85124073 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚36978984 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚319419527385232 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21546607 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5906104526848 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329519538391040 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21546863 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚325307075591384 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85124073 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149009285 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚319419510345872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329519538391040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5906104526848 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚63875067 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36978987 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚323613538925568 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚323613521886208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚63875069 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199025664§D¥6„2- _wandbruntime‚3445  _runtime‚3445E#Ã⽂ÜM =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5906694614528 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚63882388 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 196921344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚319456382413632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚329557582867968 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58532689 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3445.606871396 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329557687936512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21549412 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58532432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172017664 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚323650905292800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85133971 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5906782643712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149016359 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63882388 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚323650905292800 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚323650888253440 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5888241196956 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 trainer/global_step‚8399 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5906694614528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21549158 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5888327345542 % _timestamp‚1.7845052915435524e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149026518 P8DeviceStatsMonitor.~È% on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149026518 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6079902846976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85144128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚325344623610588 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149026518 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6079902846976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85144128 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚325344623610588 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3445.606871396 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329557687936512 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21549412 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58532432 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172017664 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚323650905292800 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85133971 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5906782643712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149016359 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚63882388 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚323650905292800 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚323650888253440 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5888241196956 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 trainer/global_step‚8399 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5906694614528 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21549158 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5888327345542 % _timestamp‚1.7845052915435524e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149026518 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5940042268672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚321263113733872 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 % _timestamp‚1.7845053096954775e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚331421794494464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚327184611642626 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64241117 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚327184508454792 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚325481857287168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚310894821839872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5921394720920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚325481840247808 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚331421899555840 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85630968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚325481857287168 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149872087 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37193487 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321263130773232 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152öíNÜoŽ×O KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5939954246656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21671454 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚325481857287168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚310894821839872 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5921394720920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚325481840247808 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚331421899555840 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85630968 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚325481857287168 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 149872087 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37193487 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚321263130773232 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5940042268672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚85630968 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58864938 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚310894846743552 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 _step‚683 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚325481840247808 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37193484 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21671712 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚317009250892288 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6114223735296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5940042268672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚321263113733872 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_start/ró w•neserved_bytes.large_pool.current‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 % _timestamp‚1.7845053096954775e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚331421794494464 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚327184611642626 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64241117 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚327184508454792 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚321300002840992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚325519206615040 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5940632356352 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37197777 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64248438 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5940720385536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21674178 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚327222159661942 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5922156820950 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85630723 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚325519206615040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37197774 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 % _timestamp‚1.7845053102591553e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5940632356352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149879161 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚317045648403456 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149879161 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85640880 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85630723 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64248440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21673923 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3464.322437004 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚310930541900800 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚310930516997120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚331459838971392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58871955 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58871697 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 trainer/global_step‚8449 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6114919804928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚325519223654400 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚317045436802048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64248438 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6115106502656 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149889320 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85640880 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64248440 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚321299985801632 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚684 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚327222056473996 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5922070672364 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149889320 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚85640880 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64248440 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚321299985801632 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚684 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚327222056473996 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5922070672364 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚321300002840992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚325519206615040 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5940632356352 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37197777 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64248438 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5940720385536 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21674178 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚327222159661942 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5922156820950 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85630723 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚325519206615040 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37197774 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 % _timestamp‚1.7845053102591553e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5940632356352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 149879161 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚317045648403456 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 149879161 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85640880 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85630723 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64248440 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21673923 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚3464.322437004 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚310930541900800 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚310930516997120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚331459838971392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58871955 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58871697 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 trainer/global_step‚8449 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6114919804928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚325519223654400 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚317045436802048 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64248438 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6115106502656¬bf‚’2’- _wandbruntime‚3464  _runtime‚3464À؆ “… 0loss/FRt2/assignment_loss‚0.9755132073685243 2loss/FBt1/assignment_loss‚0.014579127031575281 2loss/FBt2/assignment_loss‚0.010480754979506719 /loss/FBt1/detection_loss‚0.8081357475400538 trainer/global_step‚8449 2loss/SRqqt2/assignment_loss‚0.6262682128428423 /loss/FRt1/detection_loss‚0.5871857219495912 epoch‚10 &loss/total_loss‚10.038882287542743 0loss/FRt1/assignment_loss‚2.2413738206799687 /loss/FRt2/detection_loss‚1.4486663517393454 1loss/SRqqt1/detection_loss‚1.0525825839774434 /loss/FBt2/detection_loss‚0.8160600325258417 _runtime‚3464.323105415 2loss/SRqqt1/assignment_loss‚0.7821701009274731 1loss/SRqqt2/detection_loss‚0.6758666259805776 % _timestamp‚1.7845053102600932e+09 _step‚685<÷w”Š 2loss/SRqqt2/assignment_loss‚0.6262682128428423 /loss/FRt1/detection_loss‚0.5871857219495912 epoch‚10 &loss/total_loss‚10.038882287542743 0loss/FRt1/assignment_loss‚2.2413738206799687 /loss/FRt2/detection_loss‚1.4486663517393454 1loss/SRqqt1/detection_loss‚1.0525825839774434 /loss/FBt2/detection_loss‚0.8160600325258417 _runtime‚3464.323105415 2loss/SRqqt1/assignment_loss‚0.7821701009274731 1loss/SRqqt2/detection_loss‚0.6758666259805776 % _timestamp‚1.7845053102600932e+09 _step‚685 0loss/FRt2/assignment_loss‚0.9755132073685243 2loss/FBt1/assignment_loss‚0.014579127031575281 2loss/FBt2/assignment_loss‚0.010480754979506719 /loss/FBt1/detection_loss‚0.8081357475400538 trainer/global_step‚8449­¦ÆçÃl•‚(:á ÞÇõÒ´ªô¿& disk.nvme1n1.out‚10930.90576171875 disk.dm-0.in‚125.6875$ disk.nvme0n1.out‚28485.673828125' proc.memory.availableMB‚ 127.6953125! proc.memory.rssMB‚ 2874.734375) proc.memory.percent‚17.54598617553711 disk.dm-1.in‚0" disk.dm-1.out‚3196.07958984375" disk.nvme0c0n1.in‚ 113.57421875$ memory_percent‚99.22060966491699 network.recv‚985755$ disk./.usageGB‚1168.078140258789 disk.md0.out‚ 115.54296875 disk.rbd0.in‚0 cpu‚26.472745095421235 proc.cpu.threads‚17( disk./.usagePercent‚76.0839042669786( disk.nvme1c1n1.out‚10930.90576171875 disk.nvme1n1.in‚ 12.11328125 disk.md0.in‚0 disk.dm-0.out‚35975.21484375 disk.rbd0.out‚0& disk.nvme0c0n1.out‚28485.673828125 disk.nvme0n1.in‚ 113.57421875 network.sent‚ 1.11773e+07! disk.nvme1c1n1.in‚ 12.11328125xåP–‚(:… ÞÇõÒß‘£À gpu.0.gpu‚58.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚205.176' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚68.392 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$Ãׂn—‚(:ã íÇõÒã¿â¿ network.sent‚ 1.1226754e+07( disk./.usagePercent‚76.0838831466876" disk.nvme0c0n1.in‚ 113.57421875 disk.md0.in‚0 network.recv‚989943( disk.nvme1c1n1.out‚10931.74072265625 disk.md0.out‚ 116.0234375" disk.dm-0.out‚36000.6650390625! proc.memory.rssMB‚ 2874.734375 cpu‚26.47043159565279% disk.nvme0c0n1.out‚28511.52734375# disk.nvme0n1.out‚28511.52734375! disk.nvme1c1n1.in‚ 12.11328125& disk.nvme1n1.out‚10931.74072265625 disk.dm-1.in‚0" disk.dm-1.out‚3196.30029296875( proc.memory.availableMB‚ 127.91796875) proc.memory.percent‚17.54598617553711% disk./.usageGB‚1168.0778160095215 disk.nvme0n1.in‚ 113.57421875 disk.nvme1n1.in‚ 12.11328125 disk.dm-0.in‚125.6875 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚99.21925067901611 proc.cpu.threads‚17.,Ê6˜‚(:„ íÇõ҈ءÀ gpu.0.gpu‚7.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.684' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.228 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÖeðY2™- _wandbruntime‚3482  _runtime‚3482’…âžÉšÃ trainer/global_step‚8499 % _timestamp‚1.7845053283978367e+09 _runtime‚3482.460967753 _step‚686 ' lr-AdamW/pg1‚0.00010181687923512718 ' lr-AdamW/pg2‚0.00010181687923512718﫳Î›È ' lr-AdamW/pg2‚0.00010181687923512718 trainer/global_step‚8499 % _timestamp‚1.7845053283978367e+09 _runtime‚3482.460967753 _step‚686 ' lr-AdamW/pg1‚0.00010181687923512718®Jü[†2œ- _wandbruntime‚3482  _runtime‚3482Û#Á¡æÒO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚327350175648768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64607169 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6149387022848 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JDeviceStatsMonitor.on_train-hu£ò_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64607167 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59204465 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5955310344802 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21796478 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚8499 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚333324155659264 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333324050597888 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚327350175648768 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7845053283986878e+09 _step‚687 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5955224196328 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59204192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚312679579702272 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚329062044506034 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 _runtime‚3482.461709724 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5973980010496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚329061941318200 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152¯CÚÝ'ž×O ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37407987 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚318828780020224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚323106717121872 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚64607169 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64607167 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86117577 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5973891988480 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7845053283986878e+09 _step‚687 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5955224196328 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59204192 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚312679579702272 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚329062044506034 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 _runtime‚3482.461709724 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5973980010496 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚329061941318200 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚273 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚327350175648768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64607169 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6149387022848 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64607167 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59204465 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5955310344802 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21796478 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚8499 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚333324155659264 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333324050597888 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚327350175648768 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 J‹ÂÌ2Ÿ- _wandbruntime‚3482  _runtime‚3482ÒºÑ â& ÜM REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚312715274859520 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 150741963 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150752122 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64614488 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚318865179557888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5955986296358 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21798632 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59211177 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 % _timestamp‚1.7845053286539786e+09 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5955900147772 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5974570098176 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5974658127360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 150741963 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64614490 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚333362095074816 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59210906 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚8499 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _runtime‚3482.717286424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚327387524976640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 150752122 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚329099489337404 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86137632 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64614490 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚333362095074816 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59210906 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚8499 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _runtime‚3482.717286424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚327387524976640 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 150752122 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚271 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚268 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚329099489337404 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86137632 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 150752122 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64614488 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚318865179557888 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5955986296358 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21798632 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59211177 KEDeviceStatsMonitor.on_train_baþÎeDí tch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 % _timestamp‚1.7845053286539786e+09 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5955900147772 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5974570098176 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5974658127360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 150741963 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5989053671736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6007829730304 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21920912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚330939374181608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86624472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚329218494010368 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64973217 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59543399 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6007829730304 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86614329 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5989139820210 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86614329 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚329218494010368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚335226411762688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21920665 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚8549 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64973217 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59543149 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151587546 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335226306701312 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚320648737915392 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚330939477369442 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37622487 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚324950337549232 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64973219 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 _runtime‚3501.080482328 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚335226411762688 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚314464362468352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚314464337564672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0Za¹Ü'¬ÖO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚335226411762688 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚314464362468352 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚314464337564672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6184375447040 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5989053671736 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6007829730304 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21920912 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚330939374181608 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86624472 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚329218494010368 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64973217 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59543399 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6007829730304 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86614329 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5989139820210 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86614329 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚329218494010368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚335226411762688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21920665 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚8549 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64973217 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59543149 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151587546 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚335226306701312 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚320648737915392 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚330939477369442 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37622487 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚324950337549232 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64973219 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 _runtime‚3501.080482328³f Ó¶2­- _wandbruntime‚3501  _runtime‚3501ô™¶z®ÜM REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6008507840000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 trainer/global_step‚8549 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151614924 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64980540 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37626777 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86634384 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59549927 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚324987209616992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _step‚692 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151604765 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59550185 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6008507840000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151604765 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5989815771766 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚329255860377600 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37626774 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.smA)ÅÓhall_pool.current‚86148586 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86634384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚314500057625600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 % _timestamp‚1.7845053472691035e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64980538 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86624227 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚330976922200812 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚329255843338240 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚329255843338240 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚320685138514944 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚324987192577632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚314500032721920 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚335264351178240 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151614924 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚335264456246784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚3300/˜ç&¯áM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚314500032721920 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚335264351178240 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 151614924 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚335264456246784 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6008507840000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 trainer/global_step‚8549 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151614924 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64980540 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37626777 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86634384 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59549927 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚324987209616992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _step‚692 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151604765 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59550185 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6008507840000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151604765 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5989815771766 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚329255860377600 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37626774 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86634384 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚314500057625600 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 % _timestamp‚1.7845053472691035e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64980538 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86624227 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚330976922200812 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚329255843338240 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚329255843338240 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚320685138514944 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚324987192577632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717´™ P2°- _wandbruntime‚3501  _runtime‚3501mÒ¿ ±„ 2loss/SRqqt1/assignment_loss‚0.6873857319623334 2loss/SRqqt2/assignment_loss‚0.6376818141321017 1loss/SRqqt1/detection_loss‚1.2802080647656466 1loss/SRqqt2/detection_loss‚0.7341156912868662 trainer/global_step‚8549 % _timestamp‚1.7845053472700202e+09 .loss/FRt1/detection_loss‚0.597166697179465 .loss/FBt2/detection_loss‚0.642376712224984 &loss/total_loss‚10.108188232548919 epoch‚10 0loss/FRt1/assignment_loss‚2.2586191188205373 3loss/FBt1/assignment_loss‚0.0023121318251374196 3loss/FBt2/assignment_loss‚0.0056726737190342285 /loss/FBt1/detection_loss‚0.7344850174596776 _runtime‚3501.332904126 _step‚693 /loss/FRt2/assignment_loss‚1.122126939969185 /loss/FRt2/detection_loss‚1.4060376392039486ðVö1²‰ 2loss/SRqqt2/assignment_loss‚0.6376818141321017 1loss/SRqqt1/detection_loss‚1.2802080647656466 1loss/SRqqt2/detection_loss‚0.7341156912868662 trainer/global_step‚8549 % _timestamp‚1.7845053472700202e+09 .loss/FRt1/detection_loss‚0.597166697179465 .loss/FBt2/detection_loss‚0.642376712224984 &loss/total_loss‚10.108188232548919 epoch‚10 0loss/FRt1/assignment_loss‚2.2586191188205373 3loss/FBt1/assignment_loss‚0.0023121318251374196 3loss/FBt2/assignment_loss‚0.0056726737190342285 /loss/FBt1/detection_loss‚0.7344850174596776 _runtime‚3501.332904126 _step‚693 /loss/FRt2/assignment_loss‚1.122126939969185 /loss/FRt2/detection_loss‚1.4060376392039486 2loss/SRqqt1/assignment_loss‚0.6873857319623334µ'Ÿ[ðj³‚(:ß ‹ÈõÒ¥´µÀ% disk./.usageGB‚1167.7990493774414! disk.nvme1c1n1.in‚ 12.11328125! disk.dm-1.out‚3196.4931640625" disk.nvme0c0n1.in‚ 113.57421875 proc.cpu.threads‚17 network.sent‚ 1.1294208e+07 network.recv‚996884) disk./.usagePercent‚76.06572541132984& disk.nvme1c1n1.out‚10933.322265625 disk.dm-1.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 127.57421875 disk.nvme1n1.in‚ 12.11328125 disk.md0.in‚0 disk.dm-0.in‚125.6875" disk.dm-0.out‚36014.9423828125& disk.nvme0c0n1.out‚28526.279296875 disk.nvme0n1.in‚ 113.57421875# memory_percent‚99.2213487625122! proc.memory.rssMB‚ 2874.734375$ disk.nvme1n1.out‚10933.322265625 disk.md0.out‚ 116.90625 disk.rbd0.in‚0$ disk.nvme0n1.out‚28526.279296875) proc.memory.percent‚17.54598617553711 cpu‚26.4662747943261647ö)t´‚(:… ‹ÈõÒõèÀ gpu.0.gpu‚63.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.849' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.283 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z}Jš2µ- _wandbruntime‚3519  _runtime‚3519­q,ôɶà _step‚694 ' lr-AdamW/pg1‚0.00010155607999825718 ' lr-AdamW/pg2‚0.00010155607999825718 trainer/global_step‚8599 % _timestamp‚1.7845053656677983e+09 _runtime‚3519.730975668-~#úÎ·È % _timestamp‚1.7845053656677983e+09 _runtime‚3519.730975668 _step‚694 ' lr-AdamW/pg1‚0.00010155607999825718 ' lr-AdamW/pg2‚0.00010155607999825718 trainer/global_step‚8599¶؉ä2¸- _wandbruntime‚3519  _runtime‚3519ðÏ4*Ø'¹ÒO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6022883147144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22044731 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚337128562804736 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚331086812371968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65339269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59881450 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87111081 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65339269 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚337128667866112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 trainer/global_step‚8599 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65339267 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37836987 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87121224 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65339267 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚331086795332608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59881718 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6041855494144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚316249095427072 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6219363493376 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚331086812371968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚316249120330752 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6219183079936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚3519.731707749 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87121224 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87111081 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 152460493 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚322468483824128 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 152450348 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚332816807045016 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚337128667866112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6041855494144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚695 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6041767472128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚337128562804736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6022969295618 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37836984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚331086795332608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464ñ¾>º×O PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65339267 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚37836987 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87121224 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65339267 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚331086795332608 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59881718 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6041855494144 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚316249095427072 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6219363493376 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚331086812371968 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚316249120330752 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚265 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6219183079936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚3519.731707749 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87121224 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87111081 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 152460493 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚322468483824128 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 152450348 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376Ý_dÌŸ G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚332816807045016 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚337128667866112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6041855494144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚695 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6041767472128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚337128562804736 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6022969295618 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37836984 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚331086795332608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6022883147144 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22044731 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚337128562804736 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚331086812371968 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65339269 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚59881450 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87111081 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65339269 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚337128667866112 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 trainer/global_step‚8599 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0·¬þŸ®2»- _wandbruntime‚3519  _runtime‚3519'¶—hâ&¼ÜM QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65346588 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚331124178739200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87131136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚316284790584320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚337166712350208 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚331124161699840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚322504883026944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚332854355064220 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 % _timestamp‚1.7845053659250224e+09 _step‚696 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6042445581824 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚337166607281664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22047245 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322504667231232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6042533611008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚326830795965632 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚331124161699840 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6042533611008 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37841277 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152477726 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65346588 trainer/global_step‚8599 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚331124178739200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22046980 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152477726 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 _runtime‚3519.988346582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37841274 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65346590 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6220067538944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 152467567 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6023559098588 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6042445581824 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59888254þ“âç&½áM M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87131136 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚316284790584320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚337166712350208 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚331124161699840 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚322504883026944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚332854355064220 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 % _timestamp‚1.7845053659250224e+09 _step‚696 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6042445581824 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚337166607281664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22047245 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322504667231232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6042533611008 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚326830795965632 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚331124161699840 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6042533611008 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37841277 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 152477726 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65346588 trainer/global_step‚8599 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚331124178739200 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22046980 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 152477726 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 _runtime‚3519.988346582 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37841274 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65346590 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6220067538944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 152467567 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6023559098588 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6042445581824 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59888254 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65346588 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚331124178739200 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚265 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚268¸0g4q2¾- _wandbruntime‚3519  _runtime‚3519—2)¿ÿ /loss/FRt1/detection_loss‚0.5735890079930724 1loss/SRqqt2/detection_loss‚0.8014350751328864 trainer/global_step‚8599 /loss/FRt1/assignment_loss‚2.214663321344476 2loss/SRqqt2/assignment_loss‚0.7006035567917779 1loss/FBt1/assignment_loss‚0.02994068158051801 /loss/FRt2/detection_loss‚1.6228563185497795 /loss/FBt2/detection_loss‚0.6069074188514302 $ _timestamp‚1.784505365925976e+09 _runtime‚3519.988790442 _step‚697 0loss/FRt2/assignment_loss‚1.0477070494838383 1loss/SRqqt1/assignment_loss‚0.642585143657221 /loss/SRqqt1/detection_loss‚1.04612307122518 &loss/total_loss‚10.225185815202824 /loss/FBt1/detection_loss‚0.9339579196744844 epoch‚10 2loss/FBt2/assignment_loss‚0.004817250918159901µpÕ À„ epoch‚10 2loss/FBt2/assignment_loss‚0.004817250918159901 /loss/FRt1/detection_loss‚0.5735890079930724 1loss/SRqqt2/detection_loss‚0.8014350751328864 trainer/global_step‚8599 /loss/FRt1/assignment_loss‚2.214663321344476 2loss/SRqqt2/assignment_loss‚0.7006035567917779 1loss/FBt1/assignment_loss‚0.02994068158051801 /loss/FRt2/detection_loss‚1.6228563185497795 /loss/FBt2/detection_loss‚0.6069074188514302 $ _timestamp‚1.784505365925976e+09 _runtime‚3519.988790442 _step‚697 0loss/FRt2/assignment_loss‚1.0477070494838383 1loss/SRqqt1/assignment_loss‚0.642585143657221 /loss/SRqqt1/detection_loss‚1.04612307122518 &loss/total_loss‚10.225185815202824 /loss/FBt1/detection_loss‚0.9339579196744844¹“ôÍtÁ‚(:é šÈõÒµ¥÷¿! disk.dm-1.out‚3196.5908203125) proc.memory.percent‚17.54598617553711 network.recv‚ 1.001194e+06% disk./.usageGB‚1168.2259788513184 disk.dm-0.out‚36081.87109375$ memory_percent‚99.22158718109131 cpu‚26.464235479273107 disk.md0.out‚ 117.22265625 disk.dm-0.in‚125.6875) disk./.usagePercent‚76.09353387730457& disk.nvme1c1n1.out‚10990.611328125 disk.nvme1n1.in‚ 12.11328125 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 113.57421875 disk.nvme0n1.in‚ 113.57421875' disk.nvme0c0n1.out‚28536.7197265625( proc.memory.availableMB‚ 127.53515625! proc.memory.rssMB‚ 2874.734375 proc.cpu.threads‚17 network.sent‚ 1.1368305e+07% disk.nvme0n1.out‚28536.7197265625! disk.nvme1c1n1.in‚ 12.11328125$ disk.nvme1n1.out‚10990.611328125 disk.md0.in‚0 disk.dm-1.in‚0s€h›Â‚(: šÈõÒÓµ¶¿ gpu.0.gpu‚63.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚203.131' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.71033333333332 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lâ7¶2Ã- _wandbruntime‚3538  _runtime‚3538¥ØdÉÄà % _timestamp‚1.7845053843846169e+09 _runtime‚3538.448566435 _step‚698 ' lr-AdamW/pg1‚0.00010142458634877406 ' lr-AdamW/pg2‚0.00010142458634877406 trainer/global_step‚8649Ö!åÎÅÈ trainer/global_step‚8649 % _timestamp‚1.7845053843846169e+09 _runtime‚3538.448566435 _step‚698 ' lr-AdamW/pg1‚0.00010142458634877406 ' lr-AdamW/pg2‚0.00010142458634877406ºð{œi2Æ- _wandbruntime‚3538  _runtime‚3538©‚‹‰¹ ÇÑO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚324288001038848 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚339030818908160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6056712622552 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65705319 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 153313150 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38051487 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6056798771026 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65705317 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚318033853289472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65705317 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87617976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 \HDeviceStatsMonÊ”þitor.on_train_batch_start/allocated_bytes.large_pool.freed‚332955113694208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6254336551424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6075705213952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6075793235968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚8649 $ _timestamp‚1.784505384385175e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6075705213952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚334694239908424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6254147749376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚328637544325232 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6075793235968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22169880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚324288214744576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87607833 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚339030818908160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87607833 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚332955130733568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153323295 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚334694343096258 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚339030923969536 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚328637527285872 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87617976 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚332955130733568 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60221364 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚339030923969536 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _step‚699 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚332955113694208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38051484 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22170143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚318033878193152 _runtime‚3538.448630965 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65705319 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87617976 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚332955130733568 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60221364 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚339030923969536 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _step‚699 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚332955113694208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38051484 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22170143 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚318033878193152 _runtime‚3538.448630965 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65705319 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6056712622552 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚65705319 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 153313150 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38051487 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6056798771026 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚65705317 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚318033853289472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65705317 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87617976 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚332955113694208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6254336551424 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6075705213952 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚263 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6075793235968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 trainer/global_step‚8649 $ _timestamp‚1.784505384385175e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6075705213952 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚334694239908424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6254147749376 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚328637544325232 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6075793235968 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22169880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚324288214744576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87607833 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚339030818908160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87607833 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚332955130733568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152»×Ò2É- _wandbruntime‚3538  _runtime‚35389á@â&ÊÜM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22172371 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚332992480061440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 % _timestamp‚1.7845053846412158e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60228402 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22172625 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153340528 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6254848263168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚8649 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153330369 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚318069573350400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6076383323648 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65712640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65712638 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6076471352832 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60228145 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚332992497100800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚334731787927628 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87627888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6076471352832 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚332992480061440 _runtime‚3538.704551287 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65712640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚339068968453632 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚324324604116992 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚324324396709888 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153330369 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚328674399353632 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38055777 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚318069548446720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6076383323648 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87617731 ¿ê÷ÏËáM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6076383323648 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87617731 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22172371 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚332992480061440 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 % _timestamp‚1.7845053846412158e+09 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60228402 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22172625 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153340528 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6254848263168 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚8649 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 153330369 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚318069573350400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6076383323648 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65712640 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65712638 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6076471352832 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60228145 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚332992497100800 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚334731787927628 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚87627888 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6076471352832 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.free˜õÈad‚332992480061440 _runtime‚3538.704551287 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65712640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚339068968453632 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚324324604116992 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚324324396709888 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153330369 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚328674399353632 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38055777 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚318069548446720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88114728 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚330481147713232 _step‚703 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60560310 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚336571775959666 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 trainer/global_step‚8699 % _timestamp‚1.7845054027995563e+09 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚326107695485952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚330481130673872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38265984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚319818636055552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6109730977792 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6109642955776 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚334823432055808 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66071367 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6289266844672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚3556.862566007 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22294571 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6090542097960 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154186097 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚340933075011584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚334823449095168 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚340933075011584 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚334823449095168 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88114728 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚330481147713232 _step‚703 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60560310 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚336571775959666 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 trainer/global_step‚8699 % _timestamp‚1.7845054027995563e+09 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚326107695485952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚330481130673872 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38265984 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚319818636055552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6109730977792 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6109642955776 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚334823432055808 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66071367 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6289266844672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚3556.862566007 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22294571 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6090542097960¿˜Äé2Ù- _wandbruntime‚3557  _runtime‚3557›aÙ ÚÜM S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚336609220791036 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66078690 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88114483 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154193171 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22297051 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154203330 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚334860815462400 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66078690 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚334860798423040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6289781767680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚336609323978982 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22296793 \JDeviceStatsMonitor.on_train_batch_enR-)‡Äd/requested_bytes.small_pool.allocated‚ 6091304197990 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 trainer/global_step‚8699 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚340971224557056 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6110321065472 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚3557.117789381 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88124640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6110321065472 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154203330 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚326144297581056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6110409094656 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6110409094656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚326144088076800 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154193171 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38270277 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38270274 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _step‚704 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚334860798423040 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6289966368256 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚340971119488512 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚334860815462400 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60567328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60567067 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66078688 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7845054030544858e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88124640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0óâ#ç&ÛáM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60567067 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66078688 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7845054030544858e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88124640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚336609220791036 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66078690 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88114483 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 154193171 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22297051 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154203330 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚334860815462400 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66078690 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚334860798423040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6289781767680 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚336609323978982 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22296793 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6091304197990 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 trainer/global_step‚8699 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚340971224557056 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6110321065472 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚3557.117789381 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88124640 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6110321065472 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154203330 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚326144297581056 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6110409094656 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6110409094656 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚326144088076800 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154193171 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38270277 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38270274 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 _step‚704 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚334860798423040 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6289966368256 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚340971119488512 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚334860815462400 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60567328 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88611480 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚327927573737472 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 3575.67939995 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚336691767456768 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚338449105635240 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚342835331115008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22419079 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 % _timestamp‚1.7845054216164029e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88611480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6143668719616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88601337 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚327927368420352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6324179819520 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38480484 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚332324734061872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6124371573368 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6143580697600 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 155048899 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚321603393917952 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚342835436176384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚342835331115008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6124457721842 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 trainer/global_step‚8749 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88611480 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚327927573737472 ZKDeviceStatsMonitor.on_tra/O×Lin_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 3575.67939995 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚336691767456768 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚338449105635240 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚342835331115008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22419079 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 % _timestamp‚1.7845054216164029e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88611480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6143668719616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88601337 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚327927368420352 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6324179819520 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38480484 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚332324734061872 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6124371573368 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6143580697600 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 155048899 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚321603393917952 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚342835436176384 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚342835331115008 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6124457721842 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 trainer/global_step‚8749 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚342873480660480 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60906593 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6324878663168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6125047524812 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6125133673398 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7845054218721633e+09 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚336729133824000 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚332361623168992 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66444738 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66444738 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22421816 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚327963754039808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 trainer/global_step‚8749 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6144258807296 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚321639089075200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 155066132 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚88621392 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66444740 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60906340 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38484777 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚3575.935569792 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚342873480660480 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚708 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚342873375591936 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚338486653654444 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88621392 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚321639064171520 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6144346836480 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚336729116784640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6144346836480 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155066132 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 155055973 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88611235 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60906340 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38484777 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚3575.935569792 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚342873480660480 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚708 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚342873375591936 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚338486653654444 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88621392 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚321639064171520 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6144346836480 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚336729116784640 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6144346836480 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155066132 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 155055973 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88611235 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚342873480660480 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚60906593 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6324878663168 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6125047524812 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482Äu¥óÌ2ê- _wandbruntime‚3575  _runtime‚3575±[jp ëƒ 1loss/FBt1/assignment_loss‚0.03484864250873811 /loss/FBt2/detection_loss‚0.6151445757932379 %loss/total_loss‚9.406907027660923 $ _timestamp‚1.784505421873188e+09 2loss/SRqqt1/assignment_loss‚0.7783572129323961 /loss/FRt1/detection_loss‚0.5987750185022738 epoch‚10 _runtime‚3575.936024483 0loss/FRt1/assignment_loss‚2.0952413497248705 2loss/SRqqt2/assignment_loss‚0.5851406728821092 /loss/FRt2/detection_loss‚1.3347248775870661 1loss/SRqqt1/detection_loss‚1.2140783644134792 1loss/SRqqt2/detection_loss‚0.7242625172136666 _step‚709 0loss/FRt2/assignment_loss‚0.9786705299380987 2loss/FBt2/assignment_loss‚0.005944921691810934 0loss/FBt1/detection_loss‚0.44171834447317654 trainer/global_step‚8749Šo&‘ìˆ 1loss/FBt1/assignment_loss‚0.03484864250873811 /loss/FBt2/detection_loss‚0.6151445757932379 %loss/total_loss‚9.406907027660923 $ _timestamp‚1.784505421873188e+09 2loss/SRqqt1/assignment_loss‚0.7783572129323961 /loss/FRt1/detection_loss‚0.5987750185022738 epoch‚10 _runtime‚3575.936024483 0loss/FRt1/assignment_loss‚2.0952413497248705 2loss/SRqqt2/assignment_loss‚0.5851406728821092 /loss/FRt2/detection_loss‚1.3347248775870661 1loss/SRqqt1/detection_loss‚1.2140783644134792 1loss/SRqqt2/detection_loss‚0.7242625172136666 _step‚709 0loss/FRt2/assignment_loss‚0.9786705299380987 2loss/FBt2/assignment_loss‚0.005944921691810934 0loss/FBt1/detection_loss‚0.44171834447317654 trainer/global_step‚8749Ÿ+){í‚(:ð ÖÈõÒóíæ¿$ memory_percent‚99.22113418579102! proc.memory.rssMB‚ 2874.734375 network.sent‚ 1.1547156e+07! disk.nvme1c1n1.in‚ 12.11328125& disk.nvme1n1.out‚11065.35888671875 disk.md0.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 127.609375) proc.memory.percent‚17.54598617553711 cpu‚26.455873030357246 proc.cpu.threads‚17 network.recv‚ 1.016976e+06% disk./.usageGB‚1168.4643249511719 disk.nvme1n1.in‚ 12.11328125 disk.md0.out‚ 118.4453125 disk.dm-0.in‚125.6875" disk.dm-1.out‚3197.12158203125" disk.nvme0c0n1.in‚ 113.57421875( disk.nvme0c0n1.out‚28771.89111328125) disk./.usagePercent‚76.10905878203367# disk.dm-0.out‚36388.62255859375 disk.dm-1.in‚0 disk.nvme0n1.in‚ 113.57421875& disk.nvme0n1.out‚28771.89111328125( disk.nvme1c1n1.out‚11065.35888671875Ç4±œî‚(:‘ ÖÈõÒÖ«»À gpu.0.gpu‚78.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚160.753' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚53.58433333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0tL2ï- _wandbruntime‚3594  _runtime‚3594¸%gãÉðà _runtime‚3594.550296653 _step‚710 ' lr-AdamW/pg1‚0.00010102575458459456 ' lr-AdamW/pg2‚0.00010102575458459456 trainer/global_step‚8799 % _timestamp‚1.7845054404871616e+093-n]ÎñÈ trainer/global_step‚8799 % _timestamp‚1.7845054404871616e+09 _runtime‚3594.550296653 _step‚710 ' lr-AdamW/pg1‚0.00010102575458459456 ' lr-AdamW/pg2‚0.00010102575458459456Æø>‡ª2ò- _wandbruntime‚3594  _runtime‚3594^ HEóÒO N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 _step‚711 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚329747263556096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6358918779392 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚338560085818368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚344737587218432 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66803467 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚334168337449872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66803469 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚åœën“!0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 % _timestamp‚1.7845054404879987e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 155911701 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚3594.554601561 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚344737587218432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚334168354489232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22544359 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚340326641686482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6158201048776 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚344737692279808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66803467 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89098089 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89108232 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89098089 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚323388126876672 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6158287197250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚338560085818368 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6359111775744 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚340326538498648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61239074 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217900032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚344737692279808 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6177518439424 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚323388151780352 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6177518439424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚338560068779008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22544090 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482¢~ôÝ'ô×O \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚338560068779008 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38694984 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38694987 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66803469 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚329747045656064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 155901556 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚3594.554601561 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚344737587218432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚334168354489232 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22544359 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚340326641686482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6158201048776 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚344737692279808 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66803467 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89098089 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89108232 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89098089 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚323388126876672 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6158287197250 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚338560085818368 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6359111775744 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚340326538498648 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61239074 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 217900032 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚344737692279808 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6177518439424 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚323388151780352 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6177518439424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚338560068779008 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22544090 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 _step‚711 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚329747263556096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6358918779392 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚338560085818368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚344737587218432 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66803467 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚334168337449872 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66803469 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 % _timestamp‚1.7845054404879987e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 155911701 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89118144 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89118144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38699274 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66810788 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38699277 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 155918775 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚329783443948544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚8799 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66810788 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚338597435146240 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚338597452185600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚344775736763904 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚334205226556992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 155918775 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6178196549120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61245781 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22546747 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚338597435146240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22546507 _step‚712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61246024 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6359798126592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89107987 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6178284578304 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66810790 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚340364086517852 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66810790 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚329783645064192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89107987 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚323423846937600 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚334205209517632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6178196549120 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155928934 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚3594.808720596 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360/ÃY÷áM H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 155918775 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚329783443948544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚8799 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66810788 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚338597435146240 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚338597452185600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚344775736763904 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚334205226556992 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 155918775 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6178196549120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61245781 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22546747 LDDevice6>%ŽStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚338597435146240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22546507 _step‚712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61246024 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6359798126592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89107987 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6178284578304 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66810790 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚340364086517852 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66810790 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚329783645064192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89107987 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚323423846937600 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚334205209517632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6178196549120 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 155928934 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚3594.808720596 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6178284578304 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6158963148806 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89118144 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89118144 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38699274 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66810788 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38699277 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0È ¡Î2ø- _wandbruntime‚3594  _runtime‚3594 ã†Õù _step‚713 0loss/FBt1/assignment_loss‚0.1305433272597845 2loss/FBt2/assignment_loss‚0.025284088550238993 /loss/FRt1/detection_loss‚0.5638785312044926 1loss/SRqqt2/detection_loss‚0.6306223242655551 /loss/FBt1/detection_loss‚0.5997890264798815 trainer/global_step‚8799 1loss/SRqqt1/detection_loss‚0.9547662991553363 % _timestamp‚1.7845054407463486e+09 /loss/FRt1/assignment_loss‚2.174926490833362 2loss/SRqqt1/assignment_loss‚0.5496840441994164 epoch‚10 /loss/FRt2/assignment_loss‚0.946428389898662 2loss/SRqqt2/assignment_loss‚0.6781911441779088 /loss/FRt2/detection_loss‚1.5572763162841698 0loss/FBt2/detection_loss‚0.35585946876457647 %loss/total_loss‚9.167249451073385 _runtime‚3594.809217277Äüî ú† /loss/FRt1/assignment_loss‚2.174926490833362 2loss/SRqqt1/assignment_loss‚0.5496840441994164 epoch‚10 /loss/FRt2/assignment_loss‚0.946428389898662 2loss/SRqqt2/assignment_loss‚0.6781911441779088 /loss/FRt2/detection_loss‚1.5572763162841698 0loss/FBt2/detection_loss‚0.35585946876457647 %loss/total_loss‚9.167249451073385 _runtime‚3594.809217277 _step‚713 0loss/FBt1/assignment_loss‚0.1305433272597845 2loss/FBt2/assignment_loss‚0.025284088550238993 /loss/FRt1/detection_loss‚0.5638785312044926 1loss/SRqqt2/detection_loss‚0.6306223242655551 /loss/FBt1/detection_loss‚0.5997890264798815 trainer/global_step‚8799 1loss/SRqqt1/detection_loss‚0.9547662991553363 % _timestamp‚1.7845054407463486e+09É•‘ketû‚(:é åÈõÒ‰ÊÀ! proc.memory.rssMB‚ 2874.859375 network.recv‚ 1.028187e+06& disk.nvme0c0n1.out‚28898.025390625$ disk.nvme0n1.out‚28898.025390625 disk.md0.in‚0( proc.memory.availableMB‚ 127.28515625* proc.memory.percent‚17.546749114990234 network.sent‚ 1.1626012e+07( disk./.usagePercent‚76.1049833112927$ disk./.usageGB‚1168.401756286621 disk.nvme0n1.in‚ 113.57421875& disk.nvme1c1n1.out‚11139.103515625 disk.dm-0.in‚125.6875# disk.dm-0.out‚36587.63037109375" disk.dm-1.out‚3197.32568359375 disk.rbd0.out‚0 disk.md0.out‚ 118.74609375$ memory_percent‚99.22311305999756 cpu‚26.45473684893285 proc.cpu.threads‚17 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 113.57421875! disk.nvme1c1n1.in‚ 12.11328125 disk.nvme1n1.in‚ 12.11328125$ disk.nvme1n1.out‚11139.103515625 disk.dm-1.in‚0SµÞü‚(:… åÈõÒõýÀ gpu.0.gpu‚73.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚111.933' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.311 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zÇ2ý- _wandbruntime‚3614  _runtime‚3614öèø²=þ¬{ =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04530002921819687 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 jet/accuracy_2_of_5‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.028826534748077393 ,jet/accuracy_4_of_4‚0.014682539738714695 /particle/accuracy_3_of_4‚0.9981684684753418 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7165238261222839 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03318183869123459 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7538176774978638 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8055257201194763 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.724461555480957 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.524224579334259 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9821428656578064 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19290976226329803 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14619727432727814 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5827364325523376 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03341960161924362 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6409482955932617 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48500001430511475 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.028826534748077393 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6690765619277954 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6477442383766174 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15916657447814941 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7461538314819336 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 'particle/f_score‚0.6475213170051575 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 )loss/val_total_loss‚9.090769222491922 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14569422602653503 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9691704511642456 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6488098502159119 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5097846984863281 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5092984437942505 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7484102249145508 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9740267395973206 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9892321825027466 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7542111873626709 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15619607269763947 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 /particle/accuracy_4_of_4‚0.9535050988197327 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 +jet/accuracy_2_of_2‚0.14636766910552979 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9750000238418579 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.19596298038959503 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8602784276008606 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09567518532276154 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.647087574005127 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18711023032665253 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_2_of_4‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.25908106565475464 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5828073024749756 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.639532208442688 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16828130185604095 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9321563243865967 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4661126434803009 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19650641083717346 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4248994290828705 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6529805064201355 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5847204923629761 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.987684965133667 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7529556751251221 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.028189165517687798 *jet/accuracy_1_of_1‚0.3104401230812073 +particle/specificity‚0.9063578248023987 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0357142873108387 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7598781585693359 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 )jet/accuracy_1_of_3‚0.912970244884491 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12083333730697632 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8602784276008606 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 epoch‚10 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6933549046516418 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3995234966278076 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 *jet/accuracy_2_of_3‚0.5637626051902771 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6396406888961792 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8260684013366699 _runtime‚3614.241191144 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1740162968635559 % _timestamp‚1.7845054601776104e+09 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8700360655784607 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6390093564987183 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07875262945890427 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_2_of_4‚0.8140640258789062 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1753934770822525 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 .particle/accuracy_3_of_3‚0.999514102935791 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6926661729812622 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.041644442826509476 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.16088654100894928 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9945377111434937 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2964150011539459 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5566723346710205 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.188208669424057 +jet/accuracy_3_of_3‚0.06709285825490952 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0357142873108387 _step‚714 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9957804083824158 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.24270322918891907 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9963955879211426 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38250723481178284 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4448719918727875 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17453737556934357 jet/accuracy_3_of_5‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.817105233669281 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.817105233669281 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8243107795715332 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6996976733207703 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998833417892456 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02205730229616165 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7225049138069153 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37962964177131653 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8495495915412903 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.43675702810287476 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.036480341106653214 jet/accuracy_5_of_5‚0 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5193634033203125 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.804184079170227 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8898272514343262 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4204545319080353 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1473739892244339 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6862397789955139 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9952564239501953 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5261752009391785 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962481260299683 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.13484475016593933 particle/accuracy_2_of_3‚1 (particle/accuracy‚0.8497526049613953 particle/accuracy_5_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3824320435523987 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20743617415428162 *jet/accuracy_1_of_2‚0.7719336152076721 *particle/sensitivity‚0.642754077911377 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.05000000074505806 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11179091036319733 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9980574250221252 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18716099858283997 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1881226748228073 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_1_of_4‚1 jet/accuracy_4_of_5‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24403388798236847 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7580502033233643 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 particle/accuracy_4_of_5‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6260215044021606 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9951246380805969 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7validation_average_jet_accuracy‚0.35564136505126953 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6477442383766174 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.44575944542884827 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9962935447692871 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 particle/accuracy_1_of_1‚1 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5590277910232544 trainer/global_step‚8810 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.031574420630931854 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9954875707626343 +jet/accuracy_3_of_4‚0.09701317548751831 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8213369846343994 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2777777910232544 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978741407394409 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_1_of_5‚1 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.914893388748169 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5130208134651184 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7589436769485474 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_2‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03200063109397888 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01461336575448513 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17487956583499908 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.05000000074505806 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09415741264820099 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17453737556934357 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 7"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.57041335105896 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9886208772659302 particle/accuracy_1_of_3‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6237871646881104 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7470986843109131 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9870135188102722 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_5‚1鉸óÿ±{ 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17487956583499908 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.05000000074505806 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09415741264820099 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17453737556934357 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 7"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.57041335105896 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9886208772659302 particle/accuracy_1_of_3‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6237871646881104 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7470986843109131 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9870135188102722 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_5‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04530002921819687 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 jet/accuracy_2_of_5‚1 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.028826534748077393 ,jet/accuracy_4_of_4‚0.014682539738714695 /particle/accuracy_3_of_4‚0.9981684684753418 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7165238261222839 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.03318183869123459 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7538176774978638 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8055257201194763 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.724461555480957 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.524224579334259 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9821428656578064 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.19290976226329803 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14619727432727814 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5827364325523376 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03341960161924362 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6409482955932617 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48500001430511475 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.028826534748077393 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6690765619277954 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6477442383766174 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15916657447814941 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7461538314819336 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 'particle/f_score‚0.6475213170051575 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 )loss/val_total_loss‚9.090769222491922 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.14569422602653503 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 *jet/accuracy_1_of_4‚0.9691704511642456 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6488098502159119 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5097846984863281 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5092984437942505 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7484102249145508 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9740267395973206 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9892321825027466 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7542111873626709 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15619607269763947 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 /particle/accuracy_4_of_4‚0.9535050988197327 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 +jet/accuracy_2_of_2‚0.14636766910552979 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9750000238418579 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.19596298038959503 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8602784276008606 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09567518532276154 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.647087574005127 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.18711023032665253 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_2_of_4‚1 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.25908106565475464 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.5828073024749756 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.639532208442688 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.16828130185604095 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9321563243865967 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4661126434803009 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.19650641083717346 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4248994290828705 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6529805064201355 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5847204923629761 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.987684965133667 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7529556751251221 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.028189165517687798 *jet/accuracy_1_of_1‚0.3104401230812073 +particle/specificity‚0.9063578248023987 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0357142873108387 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7598781585693359 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 )jet/accuracy_1_of_3‚0.912970244884491 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.12083333730697632 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8602784276008606 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 epoch‚10 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6933549046516418 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.3995234966278076 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 *jet/accuracy_2_of_3‚0.5637””E±626051902771 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6396406888961792 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8260684013366699 _runtime‚3614.241191144 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1740162968635559 % _timestamp‚1.7845054601776104e+09 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8700360655784607 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6390093564987183 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.07875262945890427 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_2_of_4‚0.8140640258789062 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1753934770822525 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 .particle/accuracy_3_of_3‚0.999514102935791 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6926661729812622 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.041644442826509476 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.16088654100894928 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9945377111434937 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2964150011539459 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5566723346710205 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 6 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.188208669424057 +jet/accuracy_3_of_3‚0.06709285825490952 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0357142873108387 _step‚714 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 particle/accuracy_3_of_5‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9957804083824158 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.24270322918891907 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9963955879211426 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.38250723481178284 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4448719918727875 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17453737556934357 jet/accuracy_3_of_5‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 8"Purity/1FRt0SRqqt2FBt/event_purity‚0.817105233669281 6 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.817105233669281 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8243107795715332 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.6996976733207703 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998833417892456 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 :"Purity/2FRt*SRqqt*FBt/event_purity‚0.02205730229616165 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7225049138069153 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37962964177131653 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8495495915412903 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.43675702810287476 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.036480341106653214 jet/accuracy_5_of_5‚0 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5193634033203125 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.804184079170227 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8898272514343262 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4204545319080353 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1473739892244339 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6862397789955139 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9952564239501953 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5261752009391785 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962481260299683 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.13484475016593933 particle/accuracy_2_of_3‚1 (particle/accuracy‚0.8497526049613953 particle/accuracy_5_of_5‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3824320435523987 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20743617415428162 *jet/accuracy_1_of_2‚0.7719336152076721 *particle/sensitivity‚0.642754077911377 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.05000000074505806 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.11179091036319733 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9980574250221252 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.18716099858283997 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.1881226748228073 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_1_of_4‚1 jet/accuracy_4_of_5‚0 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.24403388798236847 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7580502033233643 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 particle/accuracy_4_of_5‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6260215044021606 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9951246380805969 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 7validation_average_jet_accuracy‚0.35564136505126953 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6477442383766174 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.44575944542884827 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9962935447692871 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 particle/accuracy_1_of_1‚1 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5590277910232544 trainer/global_step‚8810 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.031574420630931854 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9954875707626343 +jet/accuracy_3_of_4‚0.09701317548751831 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8213369846343994 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.2777777910232544 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978741407394409 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 jet/accuracy_1_of_5‚1 8"Purity/0FRt*SRqqt2FBt/event_purity‚0.914893388748169 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5130208134651184 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7589436769485474 particle/accuracy_1_of_2‚1 particle/accuracy_2_of_2‚1 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.03200063109397888 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01461336575448513 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1ÊÐR‘€€‚(:õ ôÈõÒŽâ˜À cpu‚26.464980674877882) disk./.usagePercent‚76.28487998665476 disk.dm-1.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 113.59765625) proc.memory.percent‚17.54155158996582 disk.nvme0n1.in‚ 113.59765625& disk.nvme0n1.out‚30822.62548828125& disk.nvme1n1.out‚11205.17724609375# memory_percent‚98.8152265548706 proc.cpu.threads‚17 network.sent‚ 1.1688611e+07% disk./.usageGB‚1171.1636199951172 disk.dm-0.in‚ 125.7109375# disk.dm-0.out‚38513.31689453125" disk.dm-1.out‚3261.25146484375( disk.nvme0c0n1.out‚30822.62548828125! disk.nvme1c1n1.in‚ 12.11328125( disk.nvme1c1n1.out‚11205.17724609375 network.recv‚ 1.032736e+06 disk.rbd0.out‚0 disk.nvme1n1.in‚ 12.11328125 disk.md0.in‚0 disk.md0.out‚ 119.2421875( proc.memory.availableMB‚ 194.11328125" proc.memory.rssMB‚ 2874.0078125í7S›‚(: ôÈõÒÁµÊÀ gpu.0.gpu‚2.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.088' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.029333333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚ð-¤2‚- _wandbruntime‚3629  _runtime‚3629×ÆbɃà _step‚715 ' lr-AdamW/pg1‚0.00010089136748918433 ' lr-AdamW/pg2‚0.00010089136748918433 trainer/global_step‚8849 % _timestamp‚1.7845054751486082e+09 _runtime‚3629.211771189&ÜIâÎ„È ' lr-AdamW/pg1‚0.00010089136748918433 ' lr-AdamW/pg2‚0.00010089136748918433 trainer/global_step‚8849 % _timestamp‚1.7845054751486082e+09 _runtime‚3629.211771189 _step‚715Ëy1G¾2…- _wandbruntime‚3629  _runtime‚3629žÍ½;Ø'†ÒO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚341401050735616 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61753859 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6212510680224 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚347632927234048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6414540563968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89762107 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22750054 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67406987 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚341401050735616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7845054751494544e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6231964520448 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89772250 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6414362247680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚336980430306224 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6231876498432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚326098332906496 _step‚716 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚343192958025808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚347632927234048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39003567 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157169092 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚3629.215555205 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚326098308002816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67406985 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22750292 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚341401067774976 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚341401067774976 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚332512670250496 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6212424531750 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6231876498432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67406987 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚347633032295424 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89762107 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67406985 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚343192854837974 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚336980447345584 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚347633032295424 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61753618 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 157179237 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808…YøÝ'‡×O L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6231876498432 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚326098332906496 _step‚716 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚343192958025808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚347632927234048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39003567 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 157169092 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚3629.215555205 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚326098308002816 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67406985 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22750292 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚341401067774976 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚341401067774976 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚332512670250496 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6212424531750 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6231876498432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67406987 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚347633032295424 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89762107 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67406985 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚343192854837974 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚336980447345584 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚347633032295424 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61753618 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 157179237 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚341401050735616 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚241 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚61753859 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6212510680224 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚347632927234048 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6414540563968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89762107 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22750054 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67406987 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚341401050735616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7845054751494544e+09 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6231964520448 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89772250 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6414362247680 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚336980430306224 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145ÌûÅ À2ˆ- _wandbruntime‚3629  _runtime‚3629î@!; ‰ÛM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89772005 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚343230402857178 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6213100483194 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚337017302373984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6232642637312 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61760305 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚343230506045124 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6232554608128 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 157186311 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61760575 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚341438417102848 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚3325492736Îê;}Ü39936 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67414308 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚326134003160064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6415056781312 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67414306 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚341438434142208 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 $ _timestamp‚1.784505475404048e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39007854 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 _step‚717 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 157196470 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚347670971710976 trainer/global_step‚8849 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚347671076779520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89782162 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚337017319413344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89782162 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22752718 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 157186311 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚341438434142208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6415245576192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22752451 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚332549059941376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6232642637312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157196470 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39007857 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚326134028063744 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22752718 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 157186311 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚341438434142208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6415245576192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚270 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22752451 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚332549059941376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6232642637312 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157196470 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39007857 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚326134028063744 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚343230402857178 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6213100483194 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚337017302373984 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6232642637312 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61760305 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚343230506045124 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6232554608128 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 157186311 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61760575 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚341438417102848 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚332549273639936 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67414308 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚326134003160064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6415056781312 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67414306 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚341438434142208 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 $ _timestamp‚1.784505475404048e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39007854 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 _step‚717 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 157196470 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚347670971710976 trainer/global_step‚8849 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚347671076779520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89782162 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚337017319413344Ͳ’Š2‹- _wandbruntime‚3629  _runtime‚3629`”®Œ 2loss/SRqqt1/assignment_loss‚0.6512761058551925 /loss/FRt1/detection_loss‚0.5935524013234602 0loss/SRqqt1/detection_loss‚1.050957657476621 %loss/total_loss‚9.658496568228738 % _timestamp‚1.7845054754049902e+09 2loss/SRqqt2/assignment_loss‚0.5861833430829566 1loss/FBt1/assignment_loss‚0.05813030414282781 1loss/FBt2/assignment_loss‚0.06625435552073419 1loss/SRqqt2/detection_loss‚0.6711404751574997 /loss/FBt2/detection_loss‚0.6162621013250258 epoch‚11 _runtime‚3629.467866995 0loss/FRt1/assignment_loss‚2.3191189566013977 0loss/FRt2/assignment_loss‚0.9591654096330915 /loss/FRt2/detection_loss‚1.5483861523015159 trainer/global_step‚8849 /loss/FBt1/detection_loss‚0.5380693058084164 _step‚718|Ô¼l † 2loss/SRqqt2/assignment_loss‚0.5861833430829566 1loss/FBt1/assignment_loss‚0.05813030414282781 1loss/FBt2/assignment_loss‚0.06625435552073419 1loss/SRqqt2/detection_loss‚0.6711404751574997 /loss/FBt2/detection_loss‚0.6162621013250258 epoch‚11 _runtime‚3629.467866995 0loss/FRt1/assignment_loss‚2.3191189566013977 0loss/FRt2/assignment_loss‚0.9591654096330915 /loss/FRt2/detection_loss‚1.5483861523015159 trainer/global_step‚8849 /loss/FBt1/detection_loss‚0.5380693058084164 _step‚718 2loss/SRqqt1/assignment_loss‚0.6512761058551925 /loss/FRt1/detection_loss‚0.5935524013234602 0loss/SRqqt1/detection_loss‚1.050957657476621 %loss/total_loss‚9.658496568228738 % _timestamp‚1.7845054754049902e+09ÎNOM|Ž‚(:ñ ƒÉõÒÖñ–À" proc.memory.rssMB‚ 2874.0078125) proc.memory.percent‚17.54155158996582 cpu‚26.462655246878327 network.recv‚ 1.037243e+06) disk./.usagePercent‚76.28489241035534! disk.nvme1c1n1.in‚ 12.11328125 disk.nvme1n1.in‚ 12.11328125% disk.nvme1n1.out‚11222.0791015625 disk.rbd0.out‚0 disk.nvme0n1.in‚ 113.59765625$ memory_percent‚98.81587028503418 network.sent‚ 1.1733295e+07% disk./.usageGB‚1171.1638107299805' disk.nvme1c1n1.out‚11222.0791015625 disk.md0.in‚0 disk.md0.out‚ 119.7265625! disk.dm-0.out‚38537.908203125 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 113.59765625 disk.dm-0.in‚ 125.7109375" disk.dm-1.out‚3277.54931640625 disk.rbd0.in‚0( disk.nvme0c0n1.out‚30847.62548828125& disk.nvme0n1.out‚30847.62548828125' proc.memory.availableMB‚ 194.0078125 proc.cpu.threads‚17X•ø‘‚(:† ƒÉõÒ§ÀÀ gpu.0.gpu‚70.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚112.158' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.386 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0GÜø{‚(:ð ’ÉõÒÿô¿ network.recv‚ 1.04806e+06) disk./.usagePercent‚76.28188065685873% disk./.usageGB‚1171.1175727844238" disk.nvme0c0n1.in‚ 113.59765625! disk.nvme1c1n1.in‚ 12.11328125( disk.nvme1c1n1.out‚11494.67138671875 disk.rbd0.out‚0" proc.memory.rssMB‚ 2874.0078125' disk.nvme0c0n1.out‚31498.3916015625 disk.nvme1n1.in‚ 12.11328125 disk.dm-0.out‚39412.56640625) proc.memory.percent‚17.54155158996582& disk.nvme1n1.out‚11494.67138671875 network.sent‚ 1.1798545e+07 disk.nvme0n1.in‚ 113.59765625 disk.dm-0.in‚ 125.7109375 disk.rbd0.in‚0( proc.memory.availableMB‚ 193.96484375% disk.nvme0n1.out‚31498.3916015625 disk.md0.in‚0 disk.md0.out‚ 120.1015625 disk.dm-1.in‚0" disk.dm-1.out‚3325.43212890625$ memory_percent‚98.81613254547119 cpu‚26.460557808630888 proc.cpu.threads‚17´ö.䛑‚(: ’ÉõÒëñ®À gpu.0.gpu‚56.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚197.465' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.82166666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0o…•Ž2’- _wandbruntime‚3648  _runtime‚3648«|Ç“Á & lr-AdamW/pg1‚0.0001007562623846176 & lr-AdamW/pg2‚0.0001007562623846176 trainer/global_step‚8899 % _timestamp‚1.7845054940041313e+09 _runtime‚3648.067244394 _step‚719é׌Ì”Æ trainer/global_step‚8899 % _timestamp‚1.7845054940041313e+09 _runtime‚3648.067244394 _step‚719 & lr-AdamW/pg1‚0.0001007562623846176 & lr-AdamW/pg2‚0.0001007562623846176Ï”Dÿ”2•- _wandbruntime‚3648  _runtime‚3648ô¨â~Ø'–ÒO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22873950 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚334332417410048 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 trainer/global_step‚8899 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6265814240256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62092279 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6246254007158 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚349535288398848 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚327883065865216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67773035 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚334332631115776 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90269002 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚343269369097216 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90269002 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚345070390889216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6265814240256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚338824050733584 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90258859 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 % _timestamp‚1.7845054940049736e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349535183337472 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚343269369097216 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22874212 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚338824033694224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚343269386136576 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158042039 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67773035 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚327883090768896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚349535183337472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚343269386136576 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6449351544832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚349535288398848 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6265902262272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39218067 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158031894 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62092014 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6265902262272 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6449540346880 _step‚720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6265814240256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62092279 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6246254007158 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚349535288398848 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚327883065865216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67773035 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚334332631115776 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90269002 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚343269369097216 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90269002 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚345070390889216 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6265814240256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚338824050733584 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90258859 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 % _timestamp‚1.7845054940049736e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349535183337472 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚343269369097216 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22874212 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚338824033694224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚343269386136576 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158042039 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67773035 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚327883090768896 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚349535183337472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚343269386136576 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6449351544832 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚349535288398848 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6265902262272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39218067 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158031894 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62092014 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6265902262272 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6449540346880 _step‚720 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6247016107188 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67780356 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222087168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 197183488 _runtime‚3648.325151615 _step‚721 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6266580379136 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90278914 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚338860922801344 % _timestamp‚1.7845054942618587e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚343306752503808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67780356 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚327918761022464 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90268757 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158059272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62098740 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158059272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22876386 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚349573227814400 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6266580379136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67780358 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6266492349952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚338860905761984 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349573332882944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62098989 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208ö”@‚ç&šáM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6266580379136 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90278914 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚338860922801344 % _timestamp‚1.7845054942618587e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚343306752503808 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67780356 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚327918761022464 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90268757 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158059272 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62098740 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158059272 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22876386 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚349573227814400 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6266580379136 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67780358 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6266492349952 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚338860905761984 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349573332882944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62098989 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚343306735464448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6247016107188 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67780356 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 222087168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 197183488 _runtime‚3648.325151615 _step‚721ÑbÇ* 2›- _wandbruntime‚3648  _runtime‚3648œÚÒ¥œ .loss/FRt1/detection_loss‚0.636173310763974 1loss/SRqqt1/detection_loss‚1.0835483779164927 1loss/SRqqt2/detection_loss‚0.6793538645851886 %loss/total_loss‚9.765045839661312 $ _timestamp‚1.784505494262769e+09 /loss/FRt1/assignment_loss‚2.156637324485928 0loss/FRt2/assignment_loss‚1.0629314411254156 2loss/SRqqt2/assignment_loss‚0.6083669570102704 2loss/FBt1/assignment_loss‚0.020564846505395574 2loss/FBt2/assignment_loss‚0.030015773810258165 /loss/FRt2/detection_loss‚1.6221611594611949 /loss/FBt1/detection_loss‚0.7341837875706994 _step‚722 epoch‚11 trainer/global_step‚8899 2loss/SRqqt1/assignment_loss‚0.7332246413902828 /loss/FBt2/detection_loss‚0.3978843550362131 _runtime‚3648.325603626Ò€^@ † /loss/FRt1/assignment_loss‚2.156637324485928 0loss/FRt2/assignment_loss‚1.0629314411254156 2loss/SRqqt2/assignment_loss‚0.6083669570102704 2loss/FBt1/assignment_loss‚0.020564846505395574 2loss/FBt2/assignment_loss‚0.030015773810258165 /loss/FRt2/detection_loss‚1.6221611594611949 /loss/FBt1/detection_loss‚0.7341837875706994 _step‚722 epoch‚11 trainer/global_step‚8899 2loss/SRqqt1/assignment_loss‚0.7332246413902828 /loss/FBt2/detection_loss‚0.3978843550362131 _runtime‚3648.325603626 .loss/FRt1/detection_loss‚0.636173310763974 1loss/SRqqt1/detection_loss‚1.0835483779164927 1loss/SRqqt2/detection_loss‚0.6793538645851886 %loss/total_loss‚9.765045839661312 $ _timestamp‚1.784505494262769e+09Òàóê×vž‚(:ë ¡ÉõÒ³Øï¿( disk./.usagePercent‚76.2818756873785 disk.md0.in‚0 disk.dm-0.in‚ 125.7109375$ memory_percent‚98.81680011749268( proc.memory.availableMB‚ 193.85546875" proc.memory.rssMB‚ 2874.0078125 disk.nvme0n1.in‚ 113.59765625& disk.nvme0n1.out‚31508.49267578125' disk.nvme1c1n1.out‚11495.4423828125 disk.nvme1n1.in‚ 12.11328125 proc.cpu.threads‚17( disk.nvme0c0n1.out‚31508.49267578125 disk.md0.out‚120.5 disk.dm-1.in‚0) proc.memory.percent‚17.54155158996582 cpu‚26.45860995580658% disk./.usageGB‚1171.1174964904785 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 113.59765625! disk.nvme1c1n1.in‚ 12.11328125% disk.nvme1n1.out‚11495.4423828125# disk.dm-0.out‚39422.37548828125! disk.dm-1.out‚3325.6298828125 network.sent‚ 1.1804478e+07 network.recv‚ 1.051114e+06•« à›Ÿ‚(: ¡ÉõÒØáÊÀ gpu.0.gpu‚51.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚129.227' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚43.07566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0춧ë* - _wandbruntime‚3666  _runtimB¯e†e‚3666[/IÇ¡Á _step‚723 & lr-AdamW/pg1‚0.0001006204414348769 & lr-AdamW/pg2‚0.0001006204414348769 trainer/global_step‚8949 % _timestamp‚1.7845055129041252e+09 _runtime‚3666.968612168ΊaeÌ¢Æ & lr-AdamW/pg1‚0.0001006204414348769 & lr-AdamW/pg2‚0.0001006204414348769 trainer/global_step‚8949 % _timestamp‚1.7845055129041252e+09 _runtime‚3666.968612168 _step‚723Ó%ÍÎ[2£- _wandbruntime‚3666  _runtime‚3666VSØ'¤ÒO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90755611 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22997795 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6299840004096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68139087 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚340667637082224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 _step‚724 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158894696 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6484286185472 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚346947823752624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6280083482566 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚351437544502272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68139085 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚340667654121584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22998041 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6299751982080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62430608 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7845055129046965e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚351437439440896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68139085 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚345137687458816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39432564 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6280169631040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚351437439440896 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158904841 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68139087 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚336152315230208 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90755611 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 trainer/global_step‚8949 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22997795 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6299840004096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68139087 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚340667637082224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 _step‚724 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 158894696 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6484286185472 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚346947823752624 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6280083482566 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚351437544502272 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68139085 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚340667654121584 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22998041 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6299751982080 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62430608 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7845055129046965e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚351437439440896 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68139085 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚345137687458816 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39432564 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6280169631040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚351437439440896 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 158904841 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68139087 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚336152315230208 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚90755611 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 trainer/global_step‚8949 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68146406 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚351475483917824 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6300518120960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62437324 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23000217 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90775666 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚346985371771940 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90775666 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚351475588986368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90765509 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90765509 trainer/global_step‚8949 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚345175070865408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39436857 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 % _timestamp‚1.7845055131611123e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158922074 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23000467 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚345175053826048 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158911915 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚329703543788544 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158911915 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚336188499423744 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚340704509149984 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68146408 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚725 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6300518120960 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6300430091776 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158922074 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6280845582596 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68146406 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62437071 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚336188715219456 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6484980538880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚725 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6300518120960 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6300430091776 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 158922074 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6280845582596 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚Ó±š¼152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68146406 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62437071 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚336188715219456 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6484980538880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68146406 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚351475483917824 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6300518120960 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62437324 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23000217 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90775666 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚346985371771940 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90775666 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚351475588986368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚90765509 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90765509 trainer/global_step‚8949 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚345175070865408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39436857 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 % _timestamp‚1.7845055131611123e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 158922074 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23000467 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚345175053826048 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 158911915 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚329703543788544 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 158911915 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚336188499423744 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚340704509149984 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68146408 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91252363 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39647067 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6519422385152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚342511240470224 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚347006022859776 _runtime‚3685.494532832 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6333777745920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚342511257509584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6313999106448 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68505137 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚331452606493696 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚331452581590016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23121423 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6333777745920 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91252363 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62768487 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚353339695544320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68505137 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68505135 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62768750 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚337972028878848 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6519239874560 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159767643 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚353339800605696 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23121683 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159757498 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353339695544320 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 _step‚728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39647064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6333689723904 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚8999 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚8999 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91252363 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39647067 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6519422385152 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚342511240470224 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚347006022859776 _runtime‚3685.494532832 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6333777745920 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚342511257509584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6313999106448 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68505137 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚331452606493696 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚331452581590016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23121423 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6333777745920 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91252363 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62768487 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚353339695544320 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68505137 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68505135 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62768750 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚337972028878848 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6519239874560 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159767643 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚353339800605696 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23121683 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 159757498 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353339695544320 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 _step‚728 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39647064 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482ØñW”2´- _wandbruntime‚3686  _runtime‚3686™O뮵ÜM trainer/global_step‚8999 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚353377845089792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39651357 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91262261 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.‚ä94$allocated‚68512458 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23123919 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68512456 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚347043372187648 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚348862804635348 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚331488276747264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚353377740021248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6334367833600 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6334455862784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68512456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 _runtime‚3686.067708239 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚338008419336704 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6314675058004 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6314588909418 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚342548129577344 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚729 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159774717 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6334367833600 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91272418 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 % _timestamp‚1.7845055320043974e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62775273 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚331488301650944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91262261 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6520117685760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68512458 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159784876 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159784876 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91272418 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚347043389227008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚338008211929600 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚348862701447402 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62775525 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 3116827512†U–ç&¶áM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68512458 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159784876 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚249 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 159784876 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91272418 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚347043389227008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚338008211929600 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚348862701447402 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62775525 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 trainer/global_step‚8999 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚353377845089792 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39651357 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91262261 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68512458 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23123919 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68512456 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚347043372187648 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚348862804635348 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚331488276747264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚353377740021248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6334367833600 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6334455862784 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68512456 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 _runtime‚3686.067708239 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚338008419336704 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6314675058004 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6314588909418 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚342548129577344 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚729 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 159774717 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6334367833600 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚91272418 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 % _timestamp‚1.7845055320043974e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62775273 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚331488301650944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91262261 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6520117685760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0Ùé†ÎÞ2·- _wandbruntime‚3686  _runtime‚3686§%™¼¸þ _step‚730 1loss/FBt1/assignment_loss‚0.05167670455906167 1loss/SRqqt1/detection_loss‚0.9811090353286738 /loss/FBt1/detection_loss‚0.8022862063468589 /loss/FBt2/detection_loss‚0.6679229335033244 epoch‚11 $ _timestamp‚1.784505532005415e+09 1loss/SRqqt1/assignment_loss‚0.624252466707266 %loss/total_loss‚9.633372518387182 0loss/FRt2/assignment_loss‚1.0096203908007195 2loss/FBt2/assignment_loss‚0.003828997785554352 /loss/FRt1/detection_loss‚0.5602389162884552 .loss/FRt2/detection_loss‚1.492440830531748 0loss/SRqqt2/detection_loss‚0.633817979979166 trainer/global_step‚8999 _runtime‚ 3686.06831491 0loss/FRt1/assignment_loss‚2.2645374398250095 2loss/SRqqt2/assignment_loss‚0.5416406167313447Kk­ ¹ƒ 0loss/FRt2/assignment_loss‚1.0096203908007195 2loss/FBt2/assignment_loss‚0.003828997785554352 /loss/FRt1/detection_loss‚0.5602389162884552 .loss/FRt2/detection_loss‚1.492440830531748 0loss/SRqqt2/detection_loss‚0.633817979979166 trainer/global_step‚8999 _runtime‚ 3686.06831491 0loss/FRt1/assignment_loss‚2.2645374398250095 2loss/SRqqt2/assignment_loss‚0.5416406167313447 _step‚730 1loss/FBt1/assignment_loss‚0.05167670455906167 1loss/SRqqt1/detection_loss‚0.9811090353286738 /loss/FBt1/detection_loss‚0.8022862063468589 /loss/FBt2/detection_loss‚0.6679229335033244 epoch‚11 $ _timestamp‚1.784505532005415e+09 1loss/SRqqt1/assignment_loss‚0.624252466707266 %loss/total_loss‚9.633372518387182Ú·Ñ>›º‚(: ¿ÉõÒÒàÔ¿ gpu.0.gpu‚57.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚204.311' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚68.10366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚350702586291606 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91759258 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7845055503404331e+09 _runtime‚3704.404645007 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 _step‚732 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚355242056709120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6367715487744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6367627465728 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91749115 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 160630445 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚350702689479440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6347742433382 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39861564 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91759258 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23245456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚344354860897584 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6367627465728 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63106780 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚355242056709120 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68871187 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91749115 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6554358354944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6347828581856 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6367715487744 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚355241951647744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚344354843858224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚9049 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63107023 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68871185 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68871187 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6554180038656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚333237364356096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160620300 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚348874341221376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68871185 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚348874341221376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚348874324182016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39861567 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚333237339452416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23245216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚339791722711040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚355241951647744 D>Dª>uM‰eviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0Æv´ªÝ'Á×O JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚243 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚355242056709120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6367715487744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6367627465728 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91749115 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 160630445 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚350702689479440 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6347742433382 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39861564 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91759258 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23245456 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚344354860897584 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6367627465728 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63106780 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚355242056709120 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68871187 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚91749115 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6554358354944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6347828581856 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6367715487744 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚355241951647744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚344354843858224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚9049 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63107023 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68871185 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68871187 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6554180038656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚333237364356096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 160620300 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚348874341221376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68871185 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚348874341221376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚348874324182016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39861567 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚333237339452416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23245216 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚339791722711040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚355241951647744 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚348874324182016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚350702586291606 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚91759258 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7845055503404331e+09 _runtime‚3704.404645007 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 _step‚732Ü­œ¹£2Â- _wandbruntime‚3704  _runtime‚3704d'fá&ÃÛM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚348911690549248 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚355280101193216 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚355279996124672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6348418384826 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160647678 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚350740134310810 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6368305575424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚344391732965344 % _timestamp‚1.7845055505955162e+09 _runtime‚ 3704.65880475 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6368393604608 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23247638 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6348504533412 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160647678 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚348911707588608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚333273034609664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68878508 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160637519 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68878508 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91769170 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6368393604608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6554879041024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚333273059513344 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68878506 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63113743 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23247886 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39865857 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚348911690549248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39865854 _step‚733 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚350740237498756 trainer/global_step‚9049 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160637519 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91759013 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6368305575424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91759013 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚355280101193216 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63113492 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482ëÏ¢æ&ÄàM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚355280101193216 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚355279996124672 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6348418384826 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 160647678 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚350740134310810 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6368305575424 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚344391732965344 % _timestamp‚1.7845055505955162e+09 _runtime‚ 3704.65880475 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6368393604608 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23247638 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6348504533412 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 160647678 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚348911707588608 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚333273034609664 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68878508 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 160637519 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68878508 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91769170 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6368393604608 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6554879041024 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚333273059513344 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68878506 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63113743 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23247886 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39865857 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚348911690549248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39865854 _step‚733 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚350740237498756 trainer/global_step‚9049 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 160637519 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91759013 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6368305575424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91759013 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚355280101193216 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63113492 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚348911690549248 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360ÝV]Ó¹2Å- _wandbruntime‚3704  _runtime‚3704®YõÕ Æ‡ 4loss/FBt2/assignment_loss‚0.00038076157560022885 /loss/FRt1/detection_loss‚0.5263721375922393 1loss/SRqqt1/detection_loss‚1.2092075317984166 0loss/FBt2/detection_loss‚0.47711646155784687 trainer/global_step‚9049 0loss/FRt2/assignment_loss‚0.9535748674167143 _runtime‚3704.659275661 0loss/FRt1/assignment_loss‚2.2639364218471028 1loss/SRqqt2/detection_loss‚0.5819423764014727 % _timestamp‚1.7845055505964568e+09 _step‚734 2loss/SRqqt1/assignment_loss‚0.7295408565182534 /loss/FRt2/detection_loss‚1.4214776242924738 /loss/FBt1/detection_loss‚0.6724251767789776 epoch‚11 %loss/total_loss‚9.472912351478957 2loss/SRqqt2/assignment_loss‚0.5796865050008505 2loss/FBt1/assignment_loss‚0.057251630699008356©Âä1ÇŒ 0loss/FRt1/assignment_loss‚2.2639364218471028 1loss/SRqqt2/detection_loss‚0.5819423764014727 % _timestamp‚1.7845055505964568e+09 _step‚734 2loss/SRqqt1/assignment_loss‚0.7295408565182534 /loss/FRt2/detection_loss‚1.4214776242924738 /loss/FBt1/detection_loss‚0.6724251767789776 epoch‚11 %loss/total_loss‚9.472912351478957 2loss/SRqqt2/assignment_loss‚0.5796865050008505 2loss/FBt1/assignment_loss‚0.057251630699008356 4loss/FBt2/assignment_loss‚0.00038076157560022885 /loss/FRt1/detection_loss‚0.5263721375922393 1loss/SRqqt1/detection_loss‚1.2092075317984166 0loss/FBt2/detection_loss‚0.47711646155784687 trainer/global_step‚9049 0loss/FRt2/assignment_loss‚0.9535748674167143 _runtime‚3704.659275661ÞPÈ$қȂ(: ÎÉõÒ¸ê¿ gpu.0.gpu‚63.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚201.304' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.10133333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÓRðÆoÉ‚(:ò ÎÉõÒ¾ÿ¿ disk.nvme1n1.in‚ 12.11328125 disk.md0.out‚ 122.5390625 disk.dm-1.in‚0& disk.nvme0n1.out‚31553.00634765625$ memory_percent‚99.21557903289795( proc.memory.availableMB‚ 128.51953125( disk./.usagePercent‚76.1948044302797& disk.nvme1n1.out‚11500.01220703125" disk.nvme0c0n1.in‚ 113.59765625( disk.nvme0c0n1.ou¯Æ»½t‚31553.00634765625 disk.nvme0n1.in‚ 113.59765625) proc.memory.percent‚17.54155158996582 cpu‚26.45218516381771 network.recv‚ 1.063567e+06 disk.md0.in‚0 disk.dm-0.in‚ 125.7109375 disk.rbd0.out‚0" proc.memory.rssMB‚ 2874.0078125 network.sent‚ 1.2003168e+07! disk.nvme1c1n1.in‚ 12.11328125" disk.dm-0.out‚39466.6767578125! disk.dm-1.out‚3326.1044921875 disk.rbd0.in‚0 proc.cpu.threads‚17% disk./.usageGB‚1169.7807350158691( disk.nvme1c1n1.out‚11500.01220703125ÃþäÆ|Ê‚(:ñ ÝÉõÒé¹÷¿ disk.md0.out‚ 123.0234375( disk.nvme0c0n1.out‚31561.48486328125& disk.nvme0n1.out‚31561.48486328125 network.recv‚ 1.067506e+06% disk./.usageGB‚1169.7808990478516& disk.nvme1n1.out‚11500.83349609375 disk.md0.in‚0 disk.dm-0.in‚ 125.7109375" disk.nvme0c0n1.in‚ 113.59765625" proc.memory.rssMB‚ 2874.0078125) proc.memory.percent‚17.54155158996582( disk./.usagePercent‚76.1948151146622 disk.nvme1n1.in‚ 12.11328125 disk.rbd0.in‚0! disk.nvme1c1n1.in‚ 12.11328125( disk.nvme1c1n1.out‚11500.83349609375( proc.memory.availableMB‚ 128.92578125 cpu‚26.450168782406305 proc.cpu.threads‚17! disk.dm-0.out‚39474.736328125 disk.dm-1.in‚0! disk.dm-1.out‚3326.3115234375 disk.rbd0.out‚0 disk.nvme0n1.in‚ 113.59765625# memory_percent‚99.2130994796753 network.sent‚ 1.2052625e+07‹jE„œË‚(:‘ ÝÉõÒ§§À gpu.0.gpu‚90.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚114.959' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.31966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”O¾>2Ì- _wandbruntime‚3723  _runtime‚3723¯a’<ÈÍ _runtime‚3723.243340958 _step‚735 ' lr-AdamW/pg1‚0.00010020870532621216 ' lr-AdamW/pg2‚0.00010020870532621216 trainer/global_step‚9099 $ _timestamp‚1.784505569180172e+09½V ÍÎÇ trainer/global_step‚9099 $ _timestamp‚1.784505569180172e+09 _runtime‚3723.243340958 _step‚735 ' lr-AdamW/pg1‚0.00010020870532621216 ' lr-AdamW/pg2‚0.00010020870532621216ßà8't2Ï- _wandbruntime‚3723  _runtime‚3723!¶ŸòØ'ÐÒO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92256010 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚357144207751168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚335022122218496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6401565207552 _runtime‚3723.247564456 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63444867 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6589142902784 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚350742659582976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 161493247 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚352580019155014 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92245867 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚341611240217600 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚350742642543616 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40076067 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚341611449729024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚346198447246224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40076064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚335022097314816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6381658057264 trainer/global_step‚9099 _step‚736 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 161483102 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚357144312812544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6401653229568 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69237235 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚350742642543616 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69237237 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63445126 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69237237 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6401653229568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6589327510528 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23368803 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚357144207751168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23369059 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚346198464285584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6401565207552 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69237235 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92256010 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚352580122342848 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚350742659582976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0ɺ*Ý'Ñ×O QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40076067 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚341611449729024 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚346198447246224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40076064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚335022097314816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚259 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6381658057264 trainer/global_step‚9099 _step‚736 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 161483102 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚357144312812544 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6401653229568 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69237235 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚350742642543616 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69237237 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63445126 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69237237 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6401653229568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6589327510528 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23368803 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚357144207751168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23369059 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚346198464285584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6401565207552 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69237235 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92256010 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚352580122342848 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚256 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚350742659582976 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92256010 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚357144207751168 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚335022122218496 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6401565207552 _runtime‚3723.247564456 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63444867 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6589142902784 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚350742659582976 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 161493247 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚352580019155014 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚92245867 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚341611240217600 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚350742642543616 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104àÔ’2Ò- _wandbruntime‚3723  _runtime‚3723Å6?`­&ÓÜM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40080354 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69244556 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69244556 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92265922 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 trainer/global_step‚9099 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23371296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚3723.501955032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚350780025950208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚335057817375744 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6589845186048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40080357 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6402243317248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92265922 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚341647840870912 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚357182357296640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161510480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚346235319313984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 161500321 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6402243317248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63451650 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69244558 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚352617670362164 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚346235336353344 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 161500321 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 >6DeviceSJ€£5tatsMonitor.on_train_batch_end/segment.all.peak‚482ÿ×-Úç&ÔáM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚357182357296640 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 161510480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚346235319313984 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 161500321 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6402243317248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63451650 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69244558 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚352617670362164 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚346235336353344 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 161500321 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40080354 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69244556 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69244556 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92265922 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 trainer/global_step‚9099 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23371296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _runtime‚3723.501955032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚350780025950208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚335057817375744 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6589845186048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40080357 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6402243317248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92265922 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚341647840870912 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330áäѾˆ2Õ- _wandbruntime‚3723  _runtime‚3723CßJb Ö† /loss/FRt1/detection_loss‚0.5805450288111964 trainer/global_step‚9099 % _timestamp‚1.7845055694395604e+09 2loss/SRqqt2/assignment_loss‚0.5713432565100476 3loss/FBt2/assignment_loss‚0.0015390347712432976 0loss/FBt2/detection_loss‚0.46824129178786456 2loss/SRqqt1/assignment_loss‚0.6836348525218996 /loss/FRt2/detection_loss‚1.7505358044993906 /loss/FBt1/detection_loss‚0.7462699225482842 _runtime‚3723.502490043 0loss/FRt1/assignment_loss‚2.1791938740790373 1loss/SRqqt1/detection_loss‚1.0718641657311516 1loss/SRqqt2/detection_loss‚0.7701148715402398 %loss/total_loss‚9.998146654400175 epoch‚11 _step‚738 /loss/FRt2/assignment_loss‚1.174437597753921 3loss/FBt1/assignment_loss‚0.0004269538458995865­ ®*׋ 0loss/FBt2/detection_loss‚0.46824129178786456 2loss/SRqqt1/assignment_loss‚0.6836348525218996 /loss/FRt2/detection_loss‚1.7505358044993906 /loss/FBt1/detection_loss‚0.7462699225482842 _runtime‚3723.502490043 0loss/FRt1/assignment_loss‚2.1791938740790373 1loss/SRqqt1/detection_loss‚1.0718641657311516 1loss/SRqqt2/detection_loss‚0.7701148715402398 %loss/total_loss‚9.998146654400175 epoch‚11 _step‚738 /loss/FRt2/assignment_loss‚1.174437597753921 3loss/FBt1/assignment_loss‚0.0004269538458995865 /loss/FRt1/detection_loss‚0.5805450288111964 trainer/global_step‚9099 % _timestamp‚1.7845055694395604e+09 2loss/SRqqt2/assignment_loss‚0.5713432565100476 3loss/FBt2/assignment_loss‚0.0015390347712432976⢛&®œØ‚(:‘ ìÉõÒíÚý¿ gpu.0.gpu‚80.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚189.662' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚63.220666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z¦€èrÙ‚(:ç ìÉõÒ Ä’À network.recv‚ 1.071248e+06 disk.dm-0.in‚ 125.7109375" disk.dm-0.out‚39479.5517578125 disk.nvme0n1.in‚ 113.59765625! disk.nvme1c1n1.in‚ 12.11328125& disk.nvme1c1n1.out‚11501.998046875 cpu‚26.45003158969787) disk./.usagePercent‚76.19483325326507$ disk./.usageGB‚1169.781177520752" disk.nvme0c0n1.in‚ 113.59765625 disk.nvme1n1.in‚ 12.11328125$ memory_percent‚99.21483993530273& proc.memory.availableMB‚ 128.640625 proc.cpu.threads‚17 disk.rbd0.in‚0% disk.nvme0c0n1.out‚31566.49609375 network.sent‚ 1.2071454e+07 disk.md0.in‚0 disk.md0.out‚ 123.62109375 disk.dm-1.in‚0! disk.dm-1.out‚3326.3818359375" proc.memory.rssMB‚ 2874.0078125 disk.rbd0.out‚0# disk.nvme0n1.out‚31566.49609375$ disk.nvme1n1.out‚11501.998046875) proc.memory.percent‚17.54155158996582‘Ô42Ú- _wandbruntime‚3742  _runtime‚3742sJ®ÓÈÛ ' lr-AdamW/pg1‚0.00010007004286439082 ' lr-AdamW/pg2‚0.00010007004286439082 trainer/global_step‚9149 $ _timestamp‚1.784505588486758e+09 _runtime‚3742.549896185 _step‚739¤F¯ÍÜÇ trainer/global_step‚9149 $ _timestamp‚1.784505588486758e+09 _runtime‚3742.549896185 _step‚739 ' lr-AdamW/pg1‚0.00010007004286439082 ' lr-AdamW/pg2‚0.00010007004286439082ãj^›2Ý- _wandbruntime‚3742  _runtime‚3742n¢´Ø'ÞÒO VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6435590971392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69603287 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚348042050634224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92752762 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚359046463854592 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7845055884876935e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23493025 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚359046463854592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63783589 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92752762 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69603285 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40290567 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚354457452018422 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162356049 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚343430943471616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6624088294400 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 trainer/global_step‚9149 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚343431150885888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6435590971392 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92742619 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚348042067673584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0Ýôï.[#ß×O VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 trainer/global_step‚9149 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚343431150885888 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6435590971392 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92742619 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚266 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚348042067673584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6435590971392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69603287 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚348042050634224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚92752762 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚359046463854592 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7845055884876935e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23493025 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚359046463854592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63783589 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚92752762 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69603285 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40290567 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚354457452018422 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162356049 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚343430943471616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6624088294400 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455ä.ƒ£—2à- _wandbruntime‚3742  _runtime‚3742h*“Áá&áÛM C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚741 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚352648327272448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63790283 _runtime‚ 3742.80843248 trainer/global_step‚9149 % _timestamp‚1.7845055887451124e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚359084613400064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40294857 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 162373282 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚359084508331520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23495429 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6624972556800 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚348078939741344 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚343467334096384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚336842550334464 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6436181059072 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92762674 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚352648327272448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63790552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23495695 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69610606 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚352648344311808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92752517 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6436269088256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6436269088256 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92762674 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚352648344311808 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚354495103225572 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6416077335642 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162373282 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69610606 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚343467547794944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152¡³•æ&âàM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚741 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚352648327272448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63790283 _runtime‚ 3742.80843248 trainer/global_step‚9149 % _timestamp‚1.7845055887451124e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚359084613400064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40294857 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 162373282 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚359084508331520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23495429 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6624972556800 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚348078939741344 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚343467334096384 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚336842550334464 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6436181059072 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚92762674 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚352648327272448 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚63790552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23495695 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69610606 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚352648344311808 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92752517 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6436269088256 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6436269088256 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92762674 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚352648344311808 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚354495103225572 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6416077335642 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162373282 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69610606 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚343467547794944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482åZô:Ý2ã- _wandbruntime‚3742  _runtime‚3742g£ö›ä€ epoch‚11 /loss/FRt1/assignment_loss‚2.123034989425406 trainer/global_step‚9149 % _timestamp‚1.7845055887460845e+09 2loss/SRqqt1/assignment_loss‚0.7675702608386544 3loss/SRqqt2/assignment_loss‚0.49314111875843764 1loss/FBt1/assignment_loss‚0.10724825558463244 .loss/FRt2/detection_loss‚1.347816285948444 .loss/FBt1/detection_loss‚0.665085780338016 _runtime‚3742.808887441 _step‚742 1loss/FBt2/assignment_loss‚0.03912977521146185 /loss/FRt1/detection_loss‚0.5454128818902497 1loss/SRqqt1/detection_loss‚1.0467623513773139 1loss/SRqqt2/detection_loss‚0.6522892674413294 /loss/FBt2/detection_loss‚0.4449186188501439 0loss/FRt2/assignment_loss‚0.7892179758612681 %loss/total_loss‚9.021627561525356]ý] å… 1loss/SRqqt2/detection_loss‚0.6522892674413294 /loss/FBt2/detection_loss‚0.4449186188501439 0loss/FRt2/assignment_loss‚0.7892179758612681 %loss/total_loss‚9.021627561525356 epoch‚11 /loss/FRt1/assignment_loss‚2.123034989425406 trainer/global_step‚9149 % _timestamp‚1.7845055887460845e+09 2loss/SRqqt1/assignment_loss‚0.7675702608386544 3loss/SRqqt2/assignment_loss‚0.49314111875843764 1loss/FBt1/assignment_loss‚0.10724825558463244 .loss/FRt2/detection_loss‚1.347816285948444 .loss/FBt1/detection_loss‚0.665085780338016 _runtime‚3742.808887441 _step‚742 1loss/FBt2/assignment_loss‚0.03912977521146185 /loss/FRt1/detection_loss‚0.5454128818902497 1loss/SRqqt1/detection_loss‚1.0467623513773139æ¦Yö‚æ‚(:÷ ûÉõÒïÒñ¿) disk./.usagePercent‚76.19477038934009# disk.dm-0.out‚39481.63916015625 disk.dm-1.in‚0 disk.nvme0n1.in‚ 113.59765625& disk.nvme0n1.out‚31568.65380859375 cpu‚26.453289718012474 proc.cpu.threads‚17 network.sent‚ 1.2119948e+07 network.recv‚ 1.075402e+06 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚31568.65380859375& disk.nvme1n1.out‚11502.40966796875( proc.memory.availableMB‚ 128.54296875 disk.dm-0.in‚ 125.7109375" disk.dm-1.out‚3326.55908203125! disk.nvme1c1n1.in‚ 12.11328125( disk.nvme1c1n1.out‚11502.40966796875 disk.md0.in‚0$ memory_percent‚99.21543598175049" proc.memory.rssMB‚ 2874.0078125% disk./.usageGB‚1169.7802124023438" disk.nvme0c0n1.in‚ 113.59765625 disk.nvme1n1.in‚ 12.11328125 disk.md0.out‚ 123.75390625) proc.memory.percent‚17.54155158996582¾"@rç‚(:… ûÉõÒç¢ï¿ gpu.0.gpu‚57.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚204.135' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚68.045 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿‘Ö2è- _wandbruntime‚3761  _runtime‚3761õ'ÎÆéÀ & lr-AdamW/pg1‚9.993067554859397e-05 & lr-AdamW/pg2‚9.993067554859397e-05 trainer/global_step‚9199 $ _timestamp‚1.784505607623611e+09 _runtime‚3761.686854954 _step‚743átF”ËêÅ $ _timestamp‚1.784505607623611e+09 _runtime‚3761.686854954 _step‚743 & lr-AdamW/pg1‚9.993067554859397e-05 & lr-AdamW/pg2‚9.993067554859397e-05 trainer/global_step‚9199çËæœ2ë- _wandbruntime‚3761  _runtime‚37612ıÆÙìÐO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69969335 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6469528713216 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6469528713216 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚349885671061584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360948719958016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69969337 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23617892 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚9199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40505064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6449230859606 $ _timestamp‚1.784505607627114e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚338591613039616 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚345250891001344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚338591637943296 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64122956 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚ 3761.69035488 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 163208706 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚356334988069664 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6449317008080 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚360948825019392 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚354479296306176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6469440691200 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6659070547456 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚354479279266816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93249514 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93249514 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚354479296306176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚265 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚69969337 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 661861171FÄê¬ý2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93239371 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚354479279266816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64123221 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93239371 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚345250683587072 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6659253058048 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚356334884881830 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93239371 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚354479279266816 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64123221 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93239371 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚345250683587072 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6659253058048 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚356334884881830 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚349885671061584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚360948719958016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69969337 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23617892 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚9199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40505064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6449230859606 $ _timestamp‚1.784505607627114e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚338591613039616 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚345250891001344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚338591637943296 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64122956 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚262 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚ 3761.69035488 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 163208706 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚356334988069664 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6449317008080 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚360948825019392 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚354479296306176 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6469440691200 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6659070547456 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚354479279266816 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93249514 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚744 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93249514 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚354479296306176 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816è'c„u2î- _wandbruntime‚3762  _runtime‚3762[‘‚wâ&ïÜM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6470118800896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚356372432901034 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163236084 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚354516662673408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163236084 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64129676 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚338627333100544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚338627308196864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6449992959636 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚349922543129344 % _timestamp‚1.7845056082026737e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚354516645634048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 163225925 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 _step‚745 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚354516662673408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93259426 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23620585 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6470118800896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69976656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚360986764434944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93249269 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93259426 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93249269 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6659954547712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚360986869503488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6449906811050 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚3762.266019202 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40509357 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0T©ðCç&ðáM PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6470118800896 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚356372432901034 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163236084 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚354516662673408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163236084 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64129676 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚338627333100544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚338627308196864 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6449992959636 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚349922543129344 % _timestamp‚1.7845056082026737e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚354516645634048 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 163225925 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 _step‚745 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚354516662673408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93259426 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23620585 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6470118800896 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69976656 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚360986764434944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93249269 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93259426 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93249269 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6659954547712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚360986869503488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚266 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6449906811050 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _runtime‚3762.266019202 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40509357 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816鿶V2ñ- _wandbruntime‚3762  _runtime‚3762‚w¡Êò€ 2loss/FBt2/assignment_loss‚0.009645725259304795 epoch‚11 /loss/FRt1/assignment_loss‚2.165715520226543 1loss/SRqqt2/assignment_loss‚0.574471136071902 2loss/FBt1/assignment_loss‚0.012955553256554866 /loss/FRt1/detection_loss‚0.5742133243577877 /loss/FBt1/detection_loss‚0.6471491486041082 trainer/global_step‚9199 _runtime‚3762.267119424 0loss/FRt2/assignment_loss‚0.8329807329937756 /loss/FRt2/detection_los'²÷¢<s‚1.5300822034144248 1loss/SRqqt1/detection_loss‚1.1437732216922927 /loss/FBt2/detection_loss‚0.4756417450794982 $loss/total_loss‚9.31741017980612 2loss/SRqqt1/assignment_loss‚0.7026829496538614 1loss/SRqqt2/detection_loss‚0.6480989191960677 $ _timestamp‚1.784505608204168e+09 _step‚746¾¾l’ ó… $ _timestamp‚1.784505608204168e+09 _step‚746 2loss/FBt2/assignment_loss‚0.009645725259304795 epoch‚11 /loss/FRt1/assignment_loss‚2.165715520226543 1loss/SRqqt2/assignment_loss‚0.574471136071902 2loss/FBt1/assignment_loss‚0.012955553256554866 /loss/FRt1/detection_loss‚0.5742133243577877 /loss/FBt1/detection_loss‚0.6471491486041082 trainer/global_step‚9199 _runtime‚3762.267119424 0loss/FRt2/assignment_loss‚0.8329807329937756 /loss/FRt2/detection_loss‚1.5300822034144248 1loss/SRqqt1/detection_loss‚1.1437732216922927 /loss/FBt2/detection_loss‚0.4756417450794982 $loss/total_loss‚9.31741017980612 2loss/SRqqt1/assignment_loss‚0.7026829496538614 1loss/SRqqt2/detection_loss‚0.6480989191960677êuxô‚(:í ŠÊõÒ›°œÀ disk.nvme1n1.in‚ 12.11328125 disk.md0.in‚0 disk.md0.out‚ 124.0703125" disk.dm-0.out‚39508.0966796875 disk.dm-1.in‚0 disk.rbd0.in‚0 network.recv‚ 1.078726e+06( disk./.usagePercent‚76.2029225731921" disk.nvme0c0n1.in‚ 113.59765625& disk.nvme0c0n1.out‚31594.912109375 disk.nvme0n1.in‚ 113.59765625$ memory_percent‚99.21445846557617" proc.memory.rssMB‚ 2874.0078125 cpu‚26.45631493572476% disk./.usageGB‚1169.9053688049316! disk.nvme1c1n1.in‚ 12.11328125& disk.nvme1n1.out‚11503.32763671875 disk.dm-0.in‚ 125.7109375 disk.rbd0.out‚0$ disk.nvme0n1.out‚31594.912109375& proc.memory.availableMB‚ 128.703125 proc.cpu.threads‚17" disk.dm-1.out‚3326.59814453125) proc.memory.percent‚17.54155158996582 network.sent‚ 1.2194099e+07( disk.nvme1c1n1.out‚11503.32763671875rÒ‚tõ‚(:… ŠÊõÒÔçÏÀ gpu.0.gpu‚59.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚204.141' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚68.047 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0AèI—›ö‚(: ™ÊõÒôÊÇ¿ gpu.0.gpu‚45.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚202.357' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.45233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fÙ8á}÷‚(:ò ™ÊõÒ†™´À' disk.nvme1c1n1.out‚11504.9248046875( proc.memory.availableMB‚ 128.42578125" proc.memory.rssMB‚ 2874.0078125 network.recv‚ 1.082928e+06" disk.nvme0c0n1.in‚ 113.59765625 disk.nvme0n1.in‚ 113.59765625 disk.md0.out‚ 125.41015625" disk.dm-0.out‚39542.8330078125 proc.cpu.threads‚17 disk.dm-1.in‚0& disk.nvme0n1.out‚31630.90576171875% disk.nvme1n1.out‚11504.9248046875# memory_percent‚99.2161512374878) proc.memory.percent‚17.54155158996582 network.sent‚ 1.2244937e+07" disk.dm-1.out‚3326.69580078125 disk.rbd0.out‚0! disk.nvme1c1n1.in‚ 12.11328125 disk.nvme1n1.in‚ 12.11328125 disk.md0.in‚0 disk.dm-0.in‚ 125.7109375 cpu‚26.457473504760106) disk./.usagePercent‚76.20646904276201$ disk./.usageGB‚1169.959815979004 disk.rbd0.in‚0( disk.nvme0c0n1.out‚31630.90576171875ülå2ø- _wandbruntime‚3781  _runtime‚3781ÓÇùÁ & lr-AdamW/pg1‚9.979060561107207e-05 & lr-AdamW/pg2‚9.979060561107207e-05 trainer/global_step‚9249 % _timestamp‚1.7845056274138763e+09 _runtime‚3781.477130268 _step‚747[ñšüÌúÆ & lr-AdamW/pg1‚9.979060561107207e-05 & lr-AdamW/pg2‚9.979060561107207e-05 trainer/global_step‚9249 % _timestamp‚1.7845056274138763e+09 _runtime‚3781.477130268 _step‚747ëˆ|I2û- _wandbruntime‚3781  _runtime‚3781{W²G×'üÑO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚93736123 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23741843 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚351729257410224 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚362850976061440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚356347614667776 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93746266 _step‚748 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6483146483488 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6694213893632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚362851081122816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70335387 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40719564 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93736123 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚356347597628416 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚356347614667776 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70335387 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70335385 trainer/global_step‚9249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚356347597628416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784505627416095e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93746266 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6503378433024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23742103 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164081653 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚362850976061440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 164071508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64461670 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6694035577344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚358212317745238 _runtime‚3781.479328512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚362851081122816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6483060335014 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6503378433024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6503466455040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64461407 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6483060335014 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6503378433024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6503466455040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚64461407 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚362850976061440 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚356347614667776 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚93746266 _step‚748 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6483146483488 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6694213893632 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚362851081122816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70335387 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40719564 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚260 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93736123 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚356347597628416 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚356347614667776 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70335387 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70335385 trainer/global_step‚9249 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚356347597628416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.784505627416095e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚93746266 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6503378433024 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23742103 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 164081653 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚362850976061440 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 164071508 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64461670 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚263 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6694035577344 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚358212317745238 _runtime‚3781.479328512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016ì‚×–2þ- _wandbruntime‚3781  _runtime‚3781TŸ [ÿÜM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70342708 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64468082 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23744228 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164088727 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93756178 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340412066059264 trainer/global_step‚9249 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70342706 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚362889125606912 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚340412090962944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚347106799885312 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚351766129477984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93746021 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40723854 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _runtime‚3781.748582665 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93756178 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6504056542720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚358249968952388 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6483822435044 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚362889020538368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164098886 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70342706 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6694914232320 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚358249865764442 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164088727 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚356384981035008 Xbj/3‡JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23744471 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚351766146517344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6504144571904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 A‘ç&€áM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40723857 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164098886 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚362889125606912 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6504144571904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23744471 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚351766146517344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6504144571904 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70342708 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64468082 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23744228 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 164088727 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93756178 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340412066059264 trainer/global_step‚9249 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70342706 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚362889125606912 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚340412090962944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚347106799885312 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚351766129477984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93746021 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40723854 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _runtime‚3781.748582665 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93756178 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6504056542720 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚358249968952388 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6483822435044 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚362889020538368 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164098886 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70342706 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6694914232320 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚358249865764442 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164088727 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚243 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚356384981035008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚364753232164864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚358215915990016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94232875 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6537316174848 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64800283 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40934067 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94243018 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚342161128764416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 164934310 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 trainer/global_step‚9299 % _timestamp‚1.7845056472484157e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6537316174848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40934064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238OÉH‹×O RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚64800283 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40934067 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94243018 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚342161128764416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 164934310 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 trainer/global_step‚9299 % _timestamp‚1.7845056472484157e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6537316174848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40934064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚364753337226240 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚342161153668096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚70701437 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70701437 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚348890358341120 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚348890153024000 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6729204673024 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6729024259584 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚353572877837584 _step‚752 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚358215933029376 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _runtime‚3801.314622209 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚364753232164864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/a»°1Î ctive_bytes.large_pool.freed‚358215915990016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94232875 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6537316174848 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164961688 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _step‚753 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚360127401815796 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚364791381710336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164951529 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70708756 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6538082313728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70708756 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23868381 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3801.579999357 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6517651910452 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚358253299396608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164961688 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6729723057152 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚358253282357248 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64806735 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚364791381710336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚348926752288768 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚342196848825344 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚360127298627850 % _timestamp‚1.7845056475166698e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0}ùRœç&ŽáM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6538082313728 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6537994284544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 164961688 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _step‚753 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚360127401815796 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚364791381710336 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 164951529 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70708756 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6538082313728 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70708756 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23868381 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚3801.579999357 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6517651910452 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚358253299396608 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 164961688 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6729723057152 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚358253282357248 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚64806735 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚364791381710336 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚348926752288768 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚342196848825344 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚360127298627850 % _timestamp‚1.7845056475166698e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40938354 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚342196823921664 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚358253299396608 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚353609749905344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚348926546978816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152ñ‘]m2- _wandbruntime‚3801  _runtime‚3801ó÷)û‚ 0loss/FRt2/assignment_loss‚0.9607908212620279 2loss/SRqqt1/assignment_loss‚0.6290089906877838 2loss/FBt1/assignment_loss‚0.007967707424064656 /loss/FRt1/detection_loss‚0.5973857397147049 1loss/SRqqt1/detection_loss‚1.1166866270974278 %loss/total_loss‚9.843510603042105 .loss/FRt2/detection_loss‚1.627518809021896 /loss/FBt1/detection_loss‚0.9423171437904239 /loss/FBt2/detection_loss‚0.6695525581151462 epoch‚11 trainer/global_step‚9299 _runtime‚3801.583566953 0loss/FRt1/assignment_loss‚2.0652397811045637 1loss/SRqqt2/assignment_loss‚0.572208555379762 3loss/FBt2/assignment_loss‚0.0036916406880224554 % _timestamp‚1.7845056475176787e+09 0loss/SRqqt2/detection_loss‚0.651142228756281 _step‚754{Í ¹ ‘‡ 3loss/FBt2/assignment_loss‚0.0036916406880224554 % _timestamp‚1.7845056475176787e+09 0loss/SRqqt2/detection_loss‚0.651142228756281 _step‚754 0loss/FRt2/assignment_loss‚0.9607908212620279 2loss/SRqqt1/assignment_loss‚0.6290089906877838 2loss/FBt1/assignment_loss‚0.007967707424064656 /loss/FRt1/detection_loss‚0.5973857397147049 1loss/SRqqt1/detection_loss‚1.1166866270974278 %loss/total_loss‚9.843510603042105 .loss/FRt2/detection_loss‚1.627518809021896 /loss/FBt1/detection_loss‚0.9423171437904239 /loss/FBt2/detection_loss‚0.6695525581151462 epoch‚11 trainer/global_step‚9299 _runtime‚3801.583566953 0loss/FRt1/assignment_loss‚2.0652397811045637 1loss/SRqqt2/assignment_loss‚0.572208555379762òB«x³r’‚(:ç ·ÊõÒĘſ" proc.memory.rssMB‚ 2874.0078125 network.recv‚ 1.090501e+06 disk.dm-0.in‚ 130.453125" disk.dm-0.out‚39997.7783203125! disk.dm-1.out‚3326.9560546875" disk.nvme0c0n1.in‚ 118.33984375& proc.memory.availableMB‚ 128.765625 cpu‚26.454289169911497 disk.rbd0.out‚0) proc.memory.percent‚17.54155158996582 proc.cpu.threads‚17 network.sent‚ 1.2312233e+07( disk./.usagePercent‚76.2185254987609 disk.nvme0n1.in‚ 118.33984375# disk.nvme0n1.out‚31954.69140625$ disk.nvme1n1.out‚11638.111328125 disk.md0.in‚0 disk.dm-1.in‚0% disk.nvme0c0n1.out‚31954.69140625% disk./.usageGB‚1170.1449127197266! disk.nvme1c1n1.in‚ 12.11328125& disk.nvme1c1n1.out‚11638.111328125 disk.nvme1n1.in‚ 12.11328125 disk.md0.out‚ 126.23828125 disk.rbd0.in‚0$ memory_percent‚99.21407699584961]¾ãŽ“‚(:ƒ ·ÊõÒø¤ý¿ gpu.0.gpu‚62.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚116.13' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.71 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yø^½2”- _wandbruntime‚3821  _runtime‚3821+·“œÇ•Á trainer/global_step‚9349 % _timestamp‚1.7845056669399092e+09 _runtime‚3821.004599606 _step‚755 & lr-AdamW/pg1‚9.950836685608842e-05 & lr-AdamW/pg2‚9.950836685608842e-05Dnœ°Ì–Æ & lr-AdamW/pg1‚9.950836685608842e-05 & lr-AdamW/pg2‚9.950836685608842e-05 trainer/global_step‚9349 % _timestamp‚1.7845056669399092e+09 _runtime‚3821.004599606 _step‚755ó Ç÷2—- _wandbruntime‚3821  _runtime‚3821UrÓN—˜ÒO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366655488268288 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71067485 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚343945911530496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6764215446528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94739770 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71067485 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6571341938688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23990213 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _step‚756 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚360084234351616 % _timestamp‚1.7845056669406035e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65138780 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6571253916672 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165807257 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94729627 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65138529 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41148567 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚360084234351616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23989965 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366655593329664 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚366655593329664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚3821.004659386 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚360084251390976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94739770 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196928512 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚361967183472054 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚eËA 6571253916672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165797112 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6764043421696 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71067487 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚350710126977024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚343945886626816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172024832 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366655488268288 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚355416464186224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6550805434304 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71067487 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6571341938688 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚350709930048512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚355416481225584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41148564 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6550719285830¯o†Ý'™×O UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚361967183472054 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6571253916672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165797112 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6764043421696 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71067487 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚350710126977024 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚343945886626816 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172024832 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚366655488268288 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚355416464186224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6550805434304 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71067487 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6571341938688 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚350709930048512 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚355416481225584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41148564 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6550719285830 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚366655488268288 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚71067485 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚343945911530496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6764215446528 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94739770 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71067485 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6571341938688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23990213 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚251 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _step‚756 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚360084234351616 % _timestamp‚1.7845056669406035e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65138780 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6571253916672 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165807257 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94729627 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65138529 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41148567 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚360084234351616 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23989965 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚366655593329664 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚366655593329664 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚3821.004659386 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚360084251390976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚94739770 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196928512ô{‹‰2š- _wandbruntime‚3821  _runtime‚3821,ñþá&›ÛM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165824490 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚360121617758208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94739525 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚343981581784064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6572020055552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6571932026368 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚362004731491258 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41152857 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚355453336253984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚360121600718848 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94749682 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6551481385860 trainer/global_step‚9349 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 $ _timestamp‚1.784505667207745e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65145559 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65145296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23992702 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚360121600718848 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165824490 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23992442 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6571932026368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚360121617758208 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165814331 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚355453353293344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚366693532745216 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94739525 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544Óɱ¬!œàM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚362004731491258 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41152857 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚355453336253984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚360121600718848 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94749682 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6551481385860 trainer/global_step‚9349 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 $ _timestamp‚1.784505667207745e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65145559 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65145296 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23992702 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚360121600718848 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165824490 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23992442 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6571932026368 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚360121617758208 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165814331 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚355453353293344 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚366693532745216 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94739525 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165824490 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚360121617758208 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚260 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚94739525 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚343981581784064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6572020055552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6571932026368 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚368557744371712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71433537 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24115040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚352529656817152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6584634909712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚345730669392896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71433537 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚357260067574224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚357260084613584 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6605279680512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65478107 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚368557849433088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚368557849433088 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 166659914 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24114781 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚361952552713216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6605279680512 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚352529862134272 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚368557744371712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65477845 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚361952552713216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41363067 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104ê»uÝ'©×O N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚361952552713216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6605279680512 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚352529862134272 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚368557744371712 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65477845 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚361952552713216 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41363067 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚368557744371712 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71433537 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24115040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚352529656817152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6584634909712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚345730669392896 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71433537 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚357260067574224 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚357260084613584 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6605279680512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65478107 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚368557849433088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚368557849433088 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 166659914 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24114781 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0øÙ°=I2ª- _wandbruntime‚3840  _runtime‚3840¬ Hu«ÜM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6605957797376 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65484661 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95236277 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚361989919080448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6799890167808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚357296956681344 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 trainer/global_step‚9399 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚361989919080448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚352566053602304 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6585224712682 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 166687292 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95246434 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚352566254717952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 166687292 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚368595893917184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65484913 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24117307 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71440858 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚345766339646464 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚363882267542612 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41367354 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6605957797376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41367357 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 58786324ÈÅŽêÛ48 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71440856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚357296939641984 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 _step‚761 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6585310861268 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚368595788848640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚361989936119808 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚345766364550144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚363882164354666 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95236277 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚3840.996744611 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6799713955840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚361989936119808 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71440858 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6605869768192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95246434 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 % _timestamp‚1.7845056869333937e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6585224712682 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 166687292 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95246434 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚352566254717952 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 166687292 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚368595893917184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65484913 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24117307 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71440858 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚345766339646464 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚363882267542612 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41367354 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6605957797376 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41367357 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71440856 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚357296939641984 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 _step‚761 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6585310861268 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚368595788848640 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚361989936119808 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚345766364550144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚363882164354666 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚95236277 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚3840.996744611 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6799713955840 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚361989936119808 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71440858 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6605869768192 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95246434 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 % _timestamp‚1.7845056869333937e+09 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65484661 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚95236277 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚361989919080448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6799890167808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚252 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚357296956681344 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 trainer/global_step‚9399 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚361989919080448 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚352566053602304 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303ù"qWS2­- _wandbruntime‚3840  _runtime‚3840la`®€ $ _timestamp‚1.784505686934487e+09 _runtime‚3840.997469082 0loss/FRt1/assignment_loss‚2.1187976500240424 /loss/FBt1/detection_loss‚0.6765725000710366 0loss/SRqqt1/detection_loss‚1.020536110392653 _step‚762 0loss/FRt2/assignment_loss‚0.9773798524871353 2loss/SRqqt2/assignment_loss‚0.5813368058008361 2loss/FBt2/assignment_loss‚0.004400559536805597 /loss/FRt1/detection_loss‚0.5657586085335655 /loss/FRt2/detection_loss‚1.4353759083896875 1loss/SRqqt2/detection_loss‚0.6977122047243433 $loss/total_loss‚9.35991542837296 epoch‚11 2loss/SRqqt1/assignment_loss‚0.6677370892756387 1loss/FBt1/assignment_loss‚0.21802955173443875 /loss/FBt2/detection_loss‚0.3962785874027759 trainer/global_step‚9399½§G¨ ¯… 2loss/SRqqt1/assignment_loss‚0.6677370892756387 1loss/FBt1/assignment_loss‚0.21802955173443875 /loss/FBt2/detection_loss‚0.3962785874027759 trainer/global_step‚9399 $ _timestamp‚1.784505686934487e+09 _runtime‚3840.997469082 0loss/FRt1/assignment_loss‚2.1187976500240424 /loss/FBt1/detection_loss‚0.6765725000710366 0loss/SRqqt1/detection_loss‚1.020536110392653 _step‚762 0loss/FRt2/assignment_loss‚0.9773798524871353 2loss/SRqqt2/assignment_loss‚0.5813368058008361 2loss/FBt2/assignment_loss‚0.004400559536805597 /loss/FRt1/detection_loss‚0.5657586085335655 /loss/FRt2/detection_loss‚1.4353759083896875 1loss/SRqqt2/detection_loss‚0.6977122047243433 $loss/total_loss‚9.35991542837296 epoch‚11ú!§k°‚(:à äÊõÒè¶é¿ disk.rbd0.out‚0# memory_percent‚99.2147445678711% proc.memory.availableMB‚ 128.65625" proc.memory.rssMB‚ 2874.0078125 network.sent‚ 1.244292e+07" disk.nvme0c0n1.in‚ 118.33984375 disk.nvme0n1.in‚ 118.33984375! disk.nvme1c1n1.in‚ 12.11328125# disk.nvme1c1n1.out‚ 11645.515625 disk.md0.out‚ 127.6953125 disk.dm-0.out‚40073.20703125! disk.dm-1.out‚3330.5849609375) disk./.usagePercent‚76.21714373478147' disk.nvme0c0n1.out‚32029.4501953125 disk.nvme1n1.in‚ 12.11328125! disk.nvme1n1.out‚ 11645.515625 disk.dm-0.in‚ 130.453125 disk.dm-1.in‚0 disk.rbd0.in‚0 proc.cpu.threads‚17) proc.memory.percent‚17.54155158996582 cpu‚26.448161429061773 network.recv‚ 1.102889e+06% disk./.usageGB‚1170.1236991882324% disk.nvme0n1.out‚32029.4501953125 disk.md0.in‚0µ×Š›±‚(: äÊõÒ÷Ðú¿ gpu.0.gpu‚73.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚192.463' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚64.15433333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Eb—¦2²- _wandbruntime‚3860  _runtime‚3860®û–ÆdzÁ & lr-AdamW/pg1‚9.922334468187361e-05 & lr-AdamW/pg2‚9.922334468187361e-05 trainer/global_step‚9449 % _timestamp‚1.7845057060493934e+09 _runtime‚3860.113196297 _step‚763]]9Ì´Æ & lr-AdamW/pg1‚9.922334468187361e-05 & lr-AdamW/pg2‚9.922334468187361e-05 trainer/global_step‚9449 % _timestamp‚1.7845057060493934e+09 _runtime‚3860.113196297 _step‚763û¾£ý¼2µ- _wandbruntime‚3860  _runtime‚3860—Ø,ÖØ'¶ÒO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 167522716 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚354349629624320 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71799587 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚370460000475136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6639129400320 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95723131 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41577567 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169927680 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚363820888114176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚347515402351616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6834032441344 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚370460105536512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65816764 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71799587 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24239200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚764 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚370460105536512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚359103688001584 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚363820888114176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚363820871074816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41577564 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751ÊÑèNË·×O UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚365722049198870 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚347515427255296 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 % _timestamp‚1.7845057060501134e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚71799585 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚363820871074816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6639129400320 trainer/global_step‚9449 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚359103670962224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6618464385120 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6618378236646 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 167532861 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95723131 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚363820888114176 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚347515402351616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6834032441344 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚370460105536512 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65816764 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71799587 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24239200 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚764 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚240 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚370460105536512 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.lardõËÂge_pool.allocated‚330 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚359103688001584 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚363820888114176 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚363820871074816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41577564 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 167522716 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚354349629624320 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71799587 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚370460000475136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6639129400320 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚95723131 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41577567 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169927680 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚359140560069344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95743186 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71806906 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41581857 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6619140336676 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚370498150020608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6639807510016 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65823525 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚363858237442048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41581854 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6619054188090 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚359140543029984 $ _timestamp‚1.784505706317099e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 _step‚765 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71806908 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6639807510016 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6639895539200 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 167539935 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚354386030252544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚370498044952064 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95743186 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚354385820748288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167550094 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24241924 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 trainer/global_step‚9449 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167550094 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚363858237442048 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚359140560069344 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95743186 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71806906 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41581857 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6619140336676 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚370498150020608 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6639807510016 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65823525 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚363858237442048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41581854 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6619054188090 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚359140543029984 $ _timestamp‚1.784505706317099e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 _step‚765 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71806908 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6639807510016 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6639895539200 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 167539935 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚354386030252544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚370498044952064 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95743186 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚354385820748288 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167550094 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24241924 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 trainer/global_step‚9449 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167550094 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚363858237442048 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72165637 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚349300185117696 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚349300160214016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚367599482062278 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚372362256578560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41792067 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24363831 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 % _timestamp‚1.7845057264451048e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6673155164160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96219883 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66155898 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6673155164160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚372362361639936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72165635 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚365689189436416 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72165637 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚349300185117696 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚349300160214016 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚367599482062278 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚372362256578560 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41792067 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24363831 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 % _timestamp‚1.7845057264451048e+09 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6673155164160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96219883 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66155898 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚237 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6673155164160€ìõÆ2Æ- _wandbruntime‚3880  _runtime‚3880qÂ2„á&ÇÛM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72172956 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6673745251840 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96229781 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66162398 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41796354 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6673833281024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6673745251840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96239938 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96229781 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168412896 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚360984146417984 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6673833281024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24366044 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚349335880274944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚365726572843008 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚365726572843008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _runtime‚3880.768558282 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚356205755815936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72172958 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚372400301055488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169920512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168402737 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚365726555803648 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 168402737 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194824192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚367637030081482 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72172958 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 _step‚769 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6869875540992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41796357 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66162642 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6652969812084 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72172956 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚367637133269428 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6652883663498 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 66186117126Û{ëdÈàM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72172956 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6673745251840 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96229781 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66162398 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41796354 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6673833281024 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6673745251840 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96239938 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96229781 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168412896 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚360984146417984 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6673833281024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24366044 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚349335880274944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚365726572843008 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚365726572843008 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _runtime‚3880.768558282 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚356205755815936 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72172958 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚372400301055488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169920512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168402737 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚365726555803648 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 168402737 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚241 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194824192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 S?DeviceStatsMonbP°;‚ itor.on_train_batch_end/requested_bytes.all.freed‚367637030081482 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚72172958 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 _step‚769 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6869875540992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41796357 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66162642 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6652969812084 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72172956 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚367637133269428 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6652883663498 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712à@x}2É- _wandbruntime‚3880  _runtime‚3880YÊ€ /loss/FRt2/detection_loss‚1.3917370216835956 1loss/SRqqt2/detection_loss‚0.6286211125463104 /loss/FBt1/detection_loss‚0.6563262781571775 epoch‚11 .loss/FRt1/assignment_loss‚2.23046830615827 0loss/FRt2/assignment_loss‚0.7684031133829805 2loss/SRqqt2/assignment_loss‚0.6472045402572821 1loss/SRqqt1/detection_loss‚1.0395607844140407 /loss/FBt2/detection_loss‚0.9193290599706498 %loss/total_loss‚9.508070170167901 trainer/global_step‚9499 _step‚770 2loss/SRqqt1/assignment_loss‚0.6384042047116086 _runtime‚3880.769047663 1loss/FBt1/assignment_loss‚0.03202270641734576 1loss/FBt2/assignment_loss‚0.06078096321387185 /loss/FRt1/detection_loss‚0.4952120792547667 % _timestamp‚1.7845057267061884e+09óoý¥ Ë… /loss/FRt2/detection_loss‚1.3917370216835956 1loss/SRqqt2/detection_loss‚0.6286211125463104 /loss/FBt1/detection_loss‚0.6563262781571775 epoch‚11 .loss/FRt1/assignment_loss‚2.23046830615827 0loss/FRt2/assignment_loss‚0.7684031133829805 2loss/SRqqt2/assignment_loss‚0.6472045402572821 1loss/SRqqt1/detection_loss‚1.0395607844140407 /loss/FBt2/detection_loss‚0.9193290599706498 %loss/total_loss‚9.508070170167901 trainer/global_step‚9499 _step‚770 2loss/SRqqt1/assignment_loss‚0.6384042047116086 _runtime‚3880.769047663 1loss/FBt1/assignment_loss‚0.03202270641734576 1loss/FBt2/assignment_loss‚0.06078096321387185 /loss/FRt1/detection_loss‚0.4952120792547667 % _timestamp‚1.7845057267061884e+09‚u›Ù|Ì‚(:ñ ‚ËõÒ¯ôâ¿ disk.dm-0.in‚ 136.921875 disk.rbd0.out‚0! disk.nvme1c1n1.in‚ 12.11328125 disk.nvme1n1.in‚ 12.11328125 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 124.80859375& disk.nvme0n1.out‚34954.27783203125' disk.nvme1c1n1.out‚12344.7666015625( proc.memory.availableMB‚ 200.05859375# proc.memory.rssMB‚ 2802.56640625 network.sent‚ 1.2580503e+07 disk.md0.in‚0# disk.dm-0.out‚43066.87060546875( disk.nvme0c0n1.out‚34954.27783203125% disk.nvme1n1.out‚12344.7666015625$ memory_percent‚98.77893924713135 proc.cpu.threads‚17 disk.dm-1.out‚ 3953.9609375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 124.80859375* proc.memory.percent‚17.105507850646973 cpu‚26.446188580739474 network.recv‚ 1.111128e+06) disk./.usagePercent‚76.40656590985856% disk./.usageGB‚1173.0318031311035 disk.md0.out‚ 131.1484375*`Í‚(:„ ‚ËõÒÌü‘À gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.762' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.254 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08–t΂(:é ‘ËõÒØþˆÀ disk.nvme1n1.in‚ 12.11328125& disk.nvme1n1.out‚12345.71142578125 disk.rbd0.out‚0& disk.nvme0n1.out‚34983.75439453125( disk.nvme1c1n1.out‚12345.71142578125 disk.md0.out‚ 131.84765625 cpu‚26.44760416682402 proc.cpu.threads‚17 network.sent‚ 1.2630748e+07) disk./.usagePercent‚76.40657112781281$ disk./.usageGB‚1173.031883239746( disk.nvme0c0n1.out‚34983.75439453125! disk.nvme1c1n1.in‚ 12.11328125 disk.md0.in‚0$ memory_percent‚98.77891540527344$ proc.memory.availableMB‚200.0625 network.recv‚ 1.114936e+06 disk.dm-0.out‚43095.73828125 disk.dm-1.out‚ 3954.0625 disk.rbd0.in‚0# proc.memory.rssMB‚ 2802.56640625* proc.memory.percent‚17.105507850646973 disk.dm-0.in‚ 139.4140625 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 127.30078125 disk.nvme0n1.in‚ 127.30078125iÍ À›Ï‚(: ‘ËõÒëó¸À gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.543' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.181000000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0S¹02Ð- _wandbruntime‚3900  _runtime‚3900ü“vÇÑÁ trainer/global_step‚9549 % _timestamp‚1.7845057461816406e+09 _runtime‚3900.246583802 _step‚771 & lr-AdamW/pg1‚9.893555734918911e-05 & lr-AdamW/pg2‚9.893555734918911e-05UBËkÌÒÆ trainer/global_step‚9549 % _timestamp‚1.7845057461816406e+09 _runtime‚3900.246583802 _step‚771 & lr-AdamW/pg1‚9.893555734918911e-05 & lr-AdamW/pg2‚9.893555734918911e-05ƒm$ z2Ó- _wandbruntime‚3900  _runtime‚3900õëWU×'ÔÑO `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚362790894777584 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚367557524837376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42006567 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6904215530496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6707092905984 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72531687 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374264617743360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374264512681984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169258465 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚351084942980096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚369477018113520 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96726778 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24488381 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _step‚772 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66494948 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96726778 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96716635 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6707004883968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚9549 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72531685 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96716635 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6707092905984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6904033019904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚369476914925686 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169248320 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _runtime‚3900.247220804 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚351084918076416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 $ _timestamp‚1.784505746184142e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1Ë«ûüÜ'ÕÖO TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72531687 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚374264512681984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66494696 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚367557524837376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚362790894777584 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚367557524837376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42006567 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6904215530496 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6707092905984 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72531687 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚374264617743360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚374264512681984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 169258465 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚351084942980096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚369477018113520 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96726778 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24488381 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _step‚772 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66494948 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96726778 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚96716635 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6707004883968 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚9549 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72531685 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96716635 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6707092905984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚252 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6904033019904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚369476914925686 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 169248320 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _runtime‚3900.247220804 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚351084918076416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 $ _timestamp‚1.784505746184142e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72539006 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66501456 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚369514462944890 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169275698 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 trainer/global_step‚9549 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚367594891204608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚362827749805984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24490602 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚358025544460800 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚358025347539456 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169275698 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6707771022848 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96736690 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚374302662227456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172017664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6686799287492 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72539008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96736690 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.curre¹Ma nt‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96726533 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169265539 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72539006 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚369514566132836 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚351120638137344 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6707682993664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24490852 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66501709 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717‘¸mRç&ØáM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96736690 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96726533 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 169265539 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72539006 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚369514566132836 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚351120638137344 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6707682993664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24490852 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66501709 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 169265539 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6707682993664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚351120613233664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72539006 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66501456 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚369514462944890 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 169275698 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 trainer/global_step‚9549 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚367594891204608 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚362827749805984 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24490602 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚358025544460800 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚358025347539456 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 169275698 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6707771022848 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96736690 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚374302662227456 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 172017664 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6686799287492 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72539008 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2…2Ù- _wandbruntime‚3900  _runtime‚3900Rðç Ú† _runtime‚3900.508066088 1loss/SRqqt2/detection_loss‚0.6447965688968613 0loss/FBt2/detection_loss‚0.36419713710421237 0loss/FRt1/assignment_loss‚2.1924259789654466 2loss/SRqqt1/assignment_loss‚0.7156110403026001 3loss/FBt2/assignment_loss‚0.0003444112055755286 trainer/global_step‚9549 % _timestamp‚1.7845057464452014e+09 _step‚774 0loss/FRt2/assignment_loss‚0.9496786271029638 /loss/FBt1/detection_loss‚0.6146355032632571 %loss/total_loss‚9.244475278940168 epoch‚11 2loss/SRqqt2/assignment_loss‚0.5702835459666101 2loss/FBt1/assignment_loss‚0.002940995265684402 /loss/FRt1/detection_loss‚0.5582933568712402 /loss/FRt2/detection_loss‚1.5790439233756983 1loss/SRqqt1/detection_loss‚1.0522241906200185W±(Û‹ 0loss/FRt2/assignment_loss‚0.9496786271029638 /loss/FBt1/detection_loss‚0.6146355032632571 %loss/total_loss‚9.244475278940168 epoch‚11 2loss/SRqqt2/assignment_loss‚0.5702835459666101 2loss/FBt1/assignment_loss‚0.002940995265684402 /loss/FRt1/detection_loss‚0.5582933568712402 /loss/FRt2/detection_loss‚1.5790439233756983 1loss/SRqqt1/detection_loss‚1.0522241906200185 _runtime‚3900.508066088 1loss/SRqqt2/detection_loss‚0.6447965688968613 0loss/FBt2/detection_loss‚0.36419713710421237 0loss/FRt1/assignment_loss‚2.1924259789654466 2loss/SRqqt1/assignment_loss‚0.7156110403026001 3loss/FBt2/assignment_loss‚0.0003444112055755286 trainer/global_step‚9549 % _timestamp‚1.7845057464452014e+09 _step‚774†SŸA&uÜ‚(:ê  ËõÒάó¿ disk.md0.out‚ 132.2578125! disk.nvme1c1n1.in‚ 12.11328125( disk.nvme1c1n1.out‚12354.38818359375# proc.memory.rssMB‚ 2802.66796875 cpu‚26.448888285457286$ disk./.usageGB‚1173.031394958496 disk.dm-0.in‚ 139.4140625 disk.dm-0.out‚43120.5! disk.dm-1.out‚3954.2763671875 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 127.30078125( disk.nvme0c0n1.out‚35000.93212890625 disk.nvme0n1.in‚ 127.30078125 disk.md0.in‚0' proc.memory.availableMB‚ 199.9296875* proc.memory.percent‚17.106127738952637 proc.cpu.threads‚17 network.recv‚ 1.118285e+06 disk.nvme1n1.in‚ 12.11328125& disk.nvme1n1.out‚12354.38818359375 disk.dm-1.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚35000.93212890625$ memory_percent‚98.77972602844238 network.sent‚ 1.2633771e+07( disk./.usagePercent‚76.4065393231393]Ú‘Ý‚(:†  ËõÒ…øÅÀ gpu.0.gpu‚76.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.933' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.311 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`7Í2Þ- _wandbruntime‚3919  _runtime‚3919¸Ð_ÒÇßÁ & lr-AdamW/pg1‚9.879063250338247e-05 & lr-AdamW/pg2‚9.879063250338247e-05 trainer/global_step‚9599 % _timestamp‚1.7845057659078333e+09 _runtime‚3919.971096626 _step‚775£à ÌàÆ trainer/global_step‚9599 % _timestamp‚1.7845057659078333e+09 _runtime‚3919.971096626 _step‚775 & lr-AdamW/pg1‚9.879063250338247e-05 & lr-AdamW/pg2‚9.879063250338247e-05‡CùñÊ2á- _wandbruntime‚3919  _runtime‚3919<×'âÑO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚369425826159616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚364634498165584 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _runtime‚ 3919.97326104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72897737 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚359808913290240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚371354347789094 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97213387 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24612445 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚352869675938816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚352869700842496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6939212447744 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6939025742848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97223530 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72897735 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6719866662870 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚376166768785408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 _step‚776 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚369425843198976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66833509 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72897737 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6741030647808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170121267 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42221064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚376166873846784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚369425843198976 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97223530 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6740942625792 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 % _timestamp‚1.7845057659100068e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6740942625792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 trainer/global_step‚9599怎 ãÖO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6740942625792 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 % _timestamp‚1.7845057659100068e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6740942625792 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 trainer/global_step‚9599 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚369425826159616 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚364634498165584 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _runtime‚ 3919.97326104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72897737 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚359808913290240 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚371354347789094 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97213387 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24612445 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚352869675938816 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚352869700842496 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6939212447744 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6939025742848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚97223530 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72897735 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6719866662870 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚úìóMN 318767104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚376166768785408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 _step‚776 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚369425843198976 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66833509 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72897737 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6741030647808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 170121267 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42221064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚376166873846784 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚369425843198976 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97223530 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ˆ¯ä Ä2ä- _wandbruntime‚3920  _runtime‚3920Ùn(dâ&åÜM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6939719965184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97223285 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 % _timestamp‚1.7845057661697555e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72905056 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97233442 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚777 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6720628762900 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66840528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚3920.233188414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6741620735488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6720542614314 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚369463192526848 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6741708764672 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚9599 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170138500 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989184 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24615171 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚371391998996244 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚364671370233344 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚369463209566208 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66840248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72905058 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚376204813262336 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 170128341 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚359845091061248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6939912954368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584# ‹ç&æáM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170138500 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989184 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24615171 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚371391998996244 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚364671370233344 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚369463209566208 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66840248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72905058 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚376204813262336 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 170128341 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚359845091061248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6939912954368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6939719965184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚277 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97223285 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 % _timestamp‚1.7845057661697555e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72905056 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97233442 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _step‚777 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6720628762900 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66840528 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _runtime‚3920.233188414 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6741620735488 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6720542614314 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚369463192526848 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6741708764672 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚9599 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448‰Û“Ž2ç- _wandbruntime‚3920  _runtime‚3920úK™¥èü 2loss/SRqqt1/assignment_loss‚0.5943728715940085 2loss/SRqqt2/assignment_loss‚0.5667302844432984 1loss/FBt1/assignment_loss‚0.04416203552292635 2loss/FBt2/assignment_loss‚0.001521800682161123 -loss/FRt2/detection_loss‚1.46600854268948 1loss/SRqqt1/detection_loss‚1.1542447092324182 $ _timestamp‚1.784505766171214e+09 _step‚778 0loss/FRt2/assignment_loss‚0.9069525022355337 /loss/FBt2/detection_loss‚0.5880261559432256 epoch‚11 -loss/FRt1/assignment_loss‚2.0899819536851 /loss/FRt1/detection_loss‚0.6217636738220406 trainer/global_step‚9599 0loss/SRqqt2/detection_loss‚0.551835367292452 /loss/FBt1/detection_loss‚0.7063442807931324 %loss/total_loss‚9.291944177935775 _runtime‚3920.234095936S½.íé epoch‚11 -loss/FRt1/assignment_loss‚2.0899819536851 /loss/FRt1/detection_loss‚0.6217636738220406 trainer/global_step‚9599 0loss/SRqqt2/detection_loss‚0.551835367292452 /loss/FBt1/detection_loss‚0.7063442807931324 %loss/total_loss‚9.291944177935775 _runtime‚3920.234095936 2loss/SRqqt1/assignment_loss‚0.5943728715940085 2loss/SRqqt2/assignment_loss‚0.5667302844432984 1loss/FBt1/assignment_loss‚0.04416203552292635 2loss/FBt2/assignment_loss‚0.001521800682161123 -loss/FRt2/detection_loss‚1.46600854268948 1loss/SRqqt1/detection_loss‚1.1542447092324182 $ _timestamp‚1.784505766171214e+09 _step‚778 0loss/FRt2/assignment_loss‚0.9069525022355337 /loss/FBt2/detection_loss‚0.5880261559432256ŠävÖrrê‚(:ç ¯ËõÒ¾š²Â disk.dm-0.in‚ 139.4140625( proc.memory.percent‚17.4593448638916 cpu‚26.453801625712774 proc.cpu.threads‚17) disk./.usagePercent‚76.40650403982963 disk.dm-1.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 127.30078125& disk.nvme0n1.out‚35002.74365234375 disk.nvme1n1.in‚ 12.11328125! disk.nvme1n1.out‚ 12354.796875# disk.dm-0.out‚43122.23876953125 network.recv‚ 1.122527e+06% disk./.usageGB‚1173.0308532714844( disk.nvme0c0n1.out‚35002.74365234375 disk.nvme0n1.in‚ 127.30078125! disk.nvme1c1n1.in‚ 12.11328125 disk.md0.in‚0 disk.md0.out‚132.4375( proc.memory.availableMB‚ 141.62109375# disk.nvme1c1n1.out‚ 12354.796875 network.sent‚ 1.2708976e+07! disk.dm-1.out‚3954.3544921875 disk.rbd0.out‚0$ memory_percent‚99.13561344146729" proc.memory.rssMB‚ 2860.5390625zyx›ë‚(: ¯ËõÒ¾¡©Â gpu.0.gpu‚42.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚185.497' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚61.83233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ+‰Å2ì- _wandbruntime‚3940  _runtime‚3940ÐWGÞð íà{ *jet/accuracy_2_of_3‚0.5434320569038391 particle/accuracy_3_of_5‚1 % _timestamp‚1.7845057862840168e+09 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7461774349212646 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_5_of_5‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2932523190975189 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9890906810760498 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15147826075553894 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6525478363037109 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 jet/accuracy_5_of_5‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03680596873164177 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8435446619987488 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4318181872367859 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 /particle/accuracy_4_of_4‚0.9407027363777161 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5186065435409546 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10000000149011612 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.15682442486286163 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 _step‚779 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7459267973899841 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7500326633453369 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45332813262939453 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.949999988079071 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7269020676612854 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16485266387462616 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.41418179869651794 *jet/accuracy_1_of_1‚0.3104427754878998 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8574562072753906 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3835327923297882 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6590946912765503 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9839434027671814 7 Purity/0FRt1SRqqt*Fë}¥Ãö1Bt/FBt_purity‚0.9921368360519409 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962481260299683 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01071486622095108 particle/accuracy_1_of_1‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1071428582072258 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15866708755493164 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12703584134578705 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3846140503883362 7"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.54527348279953 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13124999403953552 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.035149067640304565 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.019376330077648163 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16485266387462616 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7883830070495605 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2549922466278076 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6944006085395813 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995367705821991 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9985027313232422 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5900412797927856 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.021967122331261635 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.44613316655158997 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7208766341209412 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5013020634651184 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6751202344894409 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7validation_average_jet_accuracy‚0.35534438490867615 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9155736565589905 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0746544674038887 )loss/val_total_loss‚8.928436611457963 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26393353939056396 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17580857872962952 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17798759043216705 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.029912233352661133 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 )jet/accuracy_2_of_4‚0.789473831653595 *particle/specificity‚0.912808895111084 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8563912510871887 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.032269757241010666 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8100000023841858 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7867521643638611 particle/accuracy_4_of_5‚1 epoch‚11 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13715031743049622 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9870702028274536 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12543733417987823 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8778970837593079 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6960002183914185 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6358947157859802 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9947097301483154 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7883522510528564 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20536670088768005 particle/accuracy_1_of_3‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6501290202140808 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5819724798202515 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7729844450950623 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8683456182479858 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8574562072753906 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.10070161521434784 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7508801221847534 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1071428582072258 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.702521800994873 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9876542687416077 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9876542687416077 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 trainer/global_step‚9611 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_2_of_2‚0.14587178826332092 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20154142379760742 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5576741695404053 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1779845803976059 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_4‚1 /particle/accuracy_3_of_4‚0.9965986609458923 particle/accuracy_1_of_5‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.026334580034017563 +particle/sensitivity‚0.6278256177902222 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9854611754417419 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.663388729095459 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17782962322235107 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7567540407180786 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1457492858171463 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8435446619987488 particle/accuracy_1_of_4‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5486111044883728 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9741401672363281 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.656592071056366 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3194444477558136 jet/accuracy_3_of_5‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.587654709815979 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6695855855941772 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13442808389663696 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2343020737171173 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 (particle/accuracy‚0.8516089916229248 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8666667342185974 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8914567232131958 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6091208457946777 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3858024477958679 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9956080317497253 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 *jet/accuracy_1_of_4‚0.9687066078186035 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4421145021915436 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10000000149011612 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13829784095287323 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 &particle/f_score‚0.644982099533081 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.733060359954834 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6486492156982422 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6970176696777344 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4222973883152008 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9964075684547424 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6781430840492249 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_2_of_2‚1 _runtime‚3940.347645944 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08351148664951324 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.09917768090963364 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9416937232017517 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5186010599136353 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7483068704605103 +jet/accuracy_3_of_3‚0.07046699523925781 particle/accuracy_2_of_3‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5918242335319519 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8100000023841858 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9772727489471436 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9772727489471436 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4587244689464569 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7445922493934631 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4749999940395355 *jet/accuracy_1_of_3‚0.9105582237243652 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1509362906217575 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.656592071056366 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 5 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.99400794506073 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976397156715393 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.032945211976766586 particle/accuracy_3_of_3‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9464285969734192 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16420027613639832 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02960473671555519 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.993835985660553 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978741407394409 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15489517152309418 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5368208289146423 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1369834542274475 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09077876806259155 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 7"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.74539715051651 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0194295234978199 ,jet/accuracy_4_of_4‚0.016514042392373085 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8712407350540161 jet/accuracy_1_of_5‚1 jet/accuracy_4_of_5‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6907819509506226 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.021967122331261635Íh Çë=îå{ 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 8"Purity/0FRt1SRqqt0FBt/event_purity‚0.656592071056366 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3194444477558136 jet/accuracy_3_of_5‚1 6 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.587654709815979 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6695855855941772 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13442808389663696 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.2343020737171173 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 (particle/accuracy‚0.8516089916229248 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8666667342185974 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8914567232131958 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6091208457946777 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3858024477958679 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9956080317497253 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 *jet/accuracy_1_of_4‚0.9687066078186035 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4421145021915436 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10000000149011612 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13829784095287323 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 &particle/f_score‚0.644982099533081 8"Purity/*FRt*SRqqt1FBt/event_purity‚0.733060359954834 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6486492156982422 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.7142857313156128 particle/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6970176696777344 9"Purity/*FRt1SRqqt1FBt/event_purity‚0.4222973883152008 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9964075684547424 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6781430840492249 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 particle/accuracy_2_of_2‚1 _runtime‚3940.347645944 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.08351148664951324 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 +jet/accuracy_3_of_4‚0.09917768090963364 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9416937232017517 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5186010599136353 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7483068704605103 +jet/accuracy_3_of_3‚0.07046699523925781 particle/accuracy_2_of_3‚1 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5918242335319519 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8100000023841858 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9772727489471436 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9772727489471436 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.4587244689464569 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7445922493934631 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4749999940395355 *jet/accuracy_1_of_3‚0.9105582237243652 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1509362906217575 8"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.656592071056366 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 5 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.99400794506073 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9976397156715393 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.032945211976766586 particle/accuracy_3_of_3‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9464285969734192 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16420027613639832 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.02960473671555519 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.993835985660553 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978741407394409 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15489517152309418 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5368208289146423 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.1369834542274475 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.09077876806259155 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 7"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.74539715051651 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0194295234978199 ,jet/accuracy_4_of_4‚0.016514042392373085 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8712407350540161 jet/accuracy_1_of_5‚1 jet/accuracy_4_of_5‚0 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.8571428656578064 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6907819509506226 9 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.021967122331261635 *jet/accuracy_2_of_3‚0.5434320569038391 particle/accuracy_3_of_5‚1 % _timestamp‚1.7845057862840168e+09 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7461774349212646 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 particle/accuracy_5_of_5‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 particle/accuracy_1_of_2‚1 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2932523190975189 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9890906810760498 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15147826075553894 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6525478363037109 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 jet/accuracy_5_of_5‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.03680596873164177 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8435446619987488 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4318181872367859 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 /particle/accuracy_4_of_4‚0.9407027363777161 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5186065435409546 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10000000149011612 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.15682442486286163 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 _step‚779 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 + Purity/2FRt*SRqqt2FBt/FRt_purity‚0.8125 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7459267973899841 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7500326633453369 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.45332813262939453 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.949999988079071 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7269020676612854 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16485266387462616 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.41418179869651794 *jet/accuracy_1_of_1‚0.3104427754878998 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8574562072753906 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3835327923297882 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6590946912765503 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9839434027671814 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9921368360519409 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962481260299683 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.01071486622095108 particle/accuracy_1_of_1‚1 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1071428582072258 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.15866708755493164 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12703584134578705 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3846140503883362 7"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.54527348279953 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13124999403953552 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.035149067640304565 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.019376330077648163 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16485266387462616 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7883830070495605 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2549922466278076 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6944006085395813 6 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.995367705821991 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9985027313232422 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.5900412797927856 9 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.021967122331261635 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.44613316655158997 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7208766341209412 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5013020634651184 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6751202344894409 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7validation_average_jet_accuracy‚0.35534438490867615 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9155736565589905 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.0746544674038887 )loss/val_total_loss‚8.928436611457963 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26393353939056396 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17580857872962952 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17798759043216705 9 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.029912233352661133 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 )jet/accuracy_2_of_4‚0.789473831653595 *particle/specificity‚0.912808895111084 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8563912510871887 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.032269757241010666 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8100000023841858 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7867521643638611 particle/accuracy_4_of_5‚1 epoch‚11 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13715031743049622 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9870702028274536 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.12543733417987823 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ,"Purity/2FRt*SRqqt2FBt/event_purity‚0.625 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8778970837593079 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6960002183914185 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6358947157859802 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9947097301483154 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7883522510528564 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.20536670088768005 particle/accuracy_1_of_3‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6501290202140808 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5819724798202515 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7729844450950623 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8683456182479858 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8574562072753906 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.10070161521434784 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7508801221847534 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1071428582072258 8"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.702521800994873 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9876542687416077 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9876542687416077 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 trainer/global_step‚9611 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_2_of_2‚0.14587178826332092 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20154142379760742 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5576741695404053 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1779845803976059 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_4‚1 /particle/accuracy_3_of_4‚0.9965986609458923 particle/accuracy_1_of_5‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.026334580034017563 +particle/sensitivity‚0.6278256177902222 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9854611754417419 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.663388729095459 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17782962322235107 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7567540407180786 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1457492858171463 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8435446619987488 particle/accuracy_1_of_4‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5486111044883728 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9741401672363281‹'Þ—œï‚(:‘ ¾ËõÒ²ÜÉ¿ gpu.0.gpu‚43.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.416' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.138666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0城ómð‚(:â ¾ËõÒ²´À& proc.memory.availableMB‚ 193.609375* proc.memory.percent‚17.526817321777344 disk.dm-1.in‚0! disk.nvme1c1n1.in‚ 12.11328125 disk.dm-0.in‚ 139.4140625# disk.dm-0.out‚43132.43017578125 cpu‚26.46044965270098 proc.cpu.threads‚17 disk.rbd0.out‚0% disk.nvme0c0n1.out‚35012.75390625 disk.md0.out‚ 132.74609375 proc.memory.rssMB‚ 2871.59375 network.recv‚ 1.126919e+06" disk.dm-1.out‚4018.48095703125 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 127.30078125 disk.nvme0n1.in‚ 127.30078125# disk.nvme1c1n1.out‚ 12419.796875 disk.nvme1n1.in‚ 12.11328125 disk.md0.in‚0 network.sent‚ 1.2771222e+07) disk./.usagePercent‚76.40650925778387$ disk./.usageGB‚1173.030933380127# disk.nvme0n1.out‚35012.75390625! disk.nvme1n1.out‚ 12419.796875$ memory_percent‚98.81830215454102‘ž;ì2ñ- _wandbruntime‚3954  _runtime‚39549÷gÇòÁ trainer/global_step‚9649 % _timestamp‚1.7845058006974308e+09 _runtime‚3954.760581928 _step‚780 & lr-AdamW/pg1‚9.864502329595416e-05 & lr-AdamW/pg2‚9.864502329595416e-05‚²×ÌóÆ & lr-AdamW/pg1‚9.864502329595416e-05 & lr-AdamW/pg2‚9.864502329595416e-05 trainer/global_step‚9649 % _timestamp‚1.7845058006974308e+09 _runtime‚3954.760581928 _step‚780Œ 32ô- _wandbruntime‚3954  _runtime‚3954Øpñ‚àõÒO @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6774090145844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6795300684800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42529644 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚367446573982576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97877405 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174121984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97877405 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24817818 YEDeviceStats-:-øMonitor.on_train_batch_start/requested_bytes.all.allocated‚374220767316254 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚362574406904832 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6795300684800 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73501255 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚379062213862400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372266808116224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6795388706816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚367446591021936 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6774176294318 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67347208 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚379062108801024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6994723961856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 171378658 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6994549839872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _step‚781 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97887548 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171388803 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚379062108801024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372266825155584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73501253 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚362574605930496 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67347465L.HÔÝ'ö×O QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67347465 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6774090145844 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6795300684800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42529644 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚367446573982576 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚97877405 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174121984 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚97877405 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24817818 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚374220767316254 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚362574406904832 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6795300684800 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73501255 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚379062213862400 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚372266808116224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6795388706816 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚367446591021936 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6774176294318 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67347208 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚379062108801024 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6994723961856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 171378658 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6994549839872 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _step‚781 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚97887548 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 171388803 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚379062108801024 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚372266825155584 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73501253 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚362574605930496©{ik2÷- _wandbruntime‚3955  _runtime‚3955Ž<ìá&øÛM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 _step‚782 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6774766097288 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97887303 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚372304174483456 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73508574 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6995250537984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67353907 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6795978794496 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚374258315335570 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97887303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚362610802760192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42533937 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚367483463089696 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6795978794496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174114816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6774852245874 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171395877 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚379100258346496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6796066823680 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚374258212147624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3955.012147682 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚372304174483456 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6995424652800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171406036 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97897460 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚379100153277952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42533934 $ _timestamp‚1.784505800948817e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚379100258346496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚362611001778688 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚367483446050336 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330y”u†õùàM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚379100258346496 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6796066823680 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚374258212147624 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚3955.012147682 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚372304174483456 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6995424652800 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 171406036 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚97897460 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚379100153277952 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42533934 $ _timestamp‚1.784505800948817e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚379100258346496 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256ù›«êñ \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚362611001778688 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚367483446050336 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 _step‚782 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6774766097288 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚97887303 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚372304174483456 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73508574 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6995250537984 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67353907 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6795978794496 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚374258315335570 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚97887303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚362610802760192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42533937 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚367483463089696 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6795978794496 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174114816 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6774852245874 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 171395877Ž+%2ú- _wandbruntime‚3955  _runtime‚3955oéØ% ûƒ % _timestamp‚1.7845058009497058e+09 _step‚783 2loss/SRqqt1/assignment_loss‚0.5219060702916772 1loss/SRqqt2/assignment_loss‚0.638084871987646 .loss/FRt2/detection_loss‚1.653478213746486 %loss/total_loss‚8.882478173897848 0loss/FRt2/assignment_loss‚0.9708784399875279 2loss/FBt2/assignment_loss‚0.002806588464603642 /loss/FRt1/detection_loss‚0.5678304797198613 epoch‚12 _runtime‚3955.012942223 0loss/FRt1/assignment_loss‚2.1072446357979904 1loss/SRqqt1/detection_loss‚0.7545860027097524 1loss/SRqqt2/detection_loss‚0.6698198754154145 trainer/global_step‚9649 3loss/FBt1/assignment_loss‚0.0005278810421941671 /loss/FBt1/detection_loss‚0.4733713362365961 /loss/FBt2/detection_loss‚0.5219437784980983ûÉwhüˆ 3loss/FBt1/assignment_loss‚0.0005278810421941671 /loss/FBt1/detection_loss‚0.4733713362365961 /loss/FBt2/detection_loss‚0.5219437784980983 % _timestamp‚1.7845058009497058e+09 _step‚783 2loss/SRqqt1/assignment_loss‚0.5219060702916772 1loss/SRqqt2/assignment_loss‚0.638084871987646 .loss/FRt2/detection_loss‚1.653478213746486 %loss/total_loss‚8.882478173897848 0loss/FRt2/assignment_loss‚0.9708784399875279 2loss/FBt2/assignment_loss‚0.002806588464603642 /loss/FRt1/detection_loss‚0.5678304797198613 epoch‚12 _runtime‚3955.012942223 0loss/FRt1/assignment_loss‚2.1072446357979904 1loss/SRqqt1/detection_loss‚0.7545860027097524 1loss/SRqqt2/detection_loss‚0.6698198754154145 trainer/global_step‚9649ßâ˜yý‚(:î ÍËõÒ‘–í¿% disk./.usageGB‚1172.7933197021484( disk.nvme1c1n1.out‚12420.39794921875 disk.dm-0.in‚ 139.4140625$ memory_percent‚98.81489276885986( proc.memory.availableMB‚ 194.16796875 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-1.in‚0 proc.memory.rssMB‚ 2871.59375* proc.memory.percent‚17.526817321777344 cpu‚26.45890389571037 network.recv‚ 1.130939e+06 disk.rbd0.in‚0 disk.nvme0n1.in‚ 127.30078125# disk.nvme0n1.out‚35017.47265625& disk.nvme1n1.out‚12420.39794921875" disk.dm-0.out‚43136.8212890625) disk./.usagePercent‚76.39103206006503" disk.nvme0c0n1.in‚ 127.30078125% disk.nvme0c0n1.out‚35017.47265625! disk.nvme1c1n1.in‚ 12.11328125 disk.nvme1n1.in‚ 12.11328125 disk.md0.in‚0 disk.md0.out‚ 133.14453125" disk.dm-1.out‚4018.54345703125 network.sent‚ 1.2815423e+07 üµ›þ‚(: ÍËõÒö¸Ø¿ gpu.0.gpu‚64.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚194.478' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚64.82600000000001 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xF\c2ÿ- _wandbruntime‚3972  _runtime‚3972ðÞX‰Ç€Á _runtime‚3972.831365599 _step‚784 & lr-AdamW/pg1‚9.849873205913113e-05 & lr-AdamW/pg2‚9.849873205913113e-05 trainer/global_step‚9699 % _timestamp‚1.7845058187675998e+09kÔ”ÌÆ & lr-AdamW/pg2‚9.849873205913113e-05 trainer/global_step‚9699 % _timestamp‚1.7845058187675998e+09 _runtime‚3972.831365599 _step‚784 & lr-AdamW/pg1‚9.849873205913113e-05×G|]2‚- _wandbruntime‚3972  _runtime‚3972T­"Ø'ƒÒO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚380964469965824 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚242 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42744147 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6808005769726 _step‚785 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 172241460 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚3972.831466089 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6829238426624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73867305 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚380964364904448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98374157 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24942228 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚9699 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98384300 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24941989 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98374157 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7029534744064 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚374135126477824 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 172251605 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚380964364904448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67686375 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚239 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98384300 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67686133 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98384300 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚67686133 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42744147 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6808005769726 _step‚785 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 172241460 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚3972.831466089 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6829238426624 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚73867305 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚380964364904448 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98374157 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24942228 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚9699 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚98384300 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24941989 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98374157 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7029534744064 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚374135126477824 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 172251605 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚380964364904448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145‘üvU2…- _wandbruntime‚3973  _runtime‚3973.4i7r †ÜM FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚376135748198978 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚369327066477696 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6830004565504 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98394212 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7030412579840 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67692819 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚369327049438336 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 172258679 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73874626 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42748437 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_poolãˆÄ!p.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7030232173568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7845058190195088e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98384055 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24944634 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73874624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚364430747568128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚381002409381376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172258679 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _runtime‚3973.082788188 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172268838 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42748434 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172268838 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42748434 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚376135748198978 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚369327066477696 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6830004565504 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98394212 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7030412579840 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67692819 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚369327049438336 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 172258679 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73874626 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42748437 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7030232173568 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7845058190195088e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98384055 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24944634 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73874624 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚364430747568128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚381002409381376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172258679 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0’~y:+2ˆ- _wandbruntime‚3973  _runtime‚3973¼ãvQ ‰ƒ 2loss/FBt1/assignment_loss‚0.010517501803945008 /loss/FRt1/detection_loss‚0.5896209459300721 /loss/FRt2/detection_loss‚1.3034247014568083 1loss/SRqqt1/detection_loss‚1.0636689739653489 /loss/FRt1/assignment_loss‚2.083372225286439 2loss/FBt2/assignment_loss‚0.003405813648140299 epoch‚12 trainer/global_step‚9699 _runtime‚3973.083577909 2loss/SRqqt2/assignment_loss‚0.5715317900261381 1loss/SRqqt2/detection_loss‚0.5914083181586803 /loss/FBt1/detection_loss‚0.6981805199412284 % _timestamp‚1.7845058190207412e+09 2loss/SRqqt1/assignment_loss‚0.7474972730163824 /loss/FBt2/detection_loss‚0.5719725460839982 _step‚787 %loss/total_loss‚9.138460996901701 0loss/FRt2/assignment_loss‚0.9038603875845198 Á_$Šˆ %loss/total_loss‚9.138460996901701 0loss/FRt2/assignment_loss‚0.9038603875845198 2loss/FBt1/assignment_loss‚0.010517501803945008 /loss/FRt1/detection_loss‚0.5896209459300721 /loss/FRt2/detection_loss‚1.3034247014568083 1loss/SRqqt1/detection_loss‚1.0636689739653489 /loss/FRt1/assignment_loss‚2.083372225286439 2loss/FBt2/assignment_loss‚0.003405813648140299 epoch‚12 trainer/global_step‚9699 _runtime‚3973.083577909 2loss/SRqqt2/assignment_loss‚0.5715317900261381 1loss/SRqqt2/detection_loss‚0.5914083181586803 /loss/FBt1/detection_loss‚0.6981805199412284 % _timestamp‚1.7845058190207412e+09 2loss/SRqqt1/assignment_loss‚0.7474972730163824 /loss/FBt2/detection_loss‚0.5719725460839982 _step‚787“_k—vq‹‚(:æ ÜËõÒ¯Òø¿ disk.nvme1n1.in‚ 12.11328125 disk.md0.in‚0( proc.memory.availableMB‚ 194.16796875* proc.memory.percent‚17.526817321777344 proc.cpu.threads‚17 network.sent‚ 1.2877698e+07 network.recv‚ 1.135232e+06 disk.dm-0.in‚ 139.4140625 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme1c1n1.in‚ 12.11328125 disk.md0.out‚133.625( disk.nvme0c0n1.out‚35031.97509765625$ disk.nvme1c1n1.out‚ 12487.2265625" disk.nvme1n1.out‚ 12487.2265625( disk./.usagePercent‚76.3910482108758# disk.dm-0.out‚43153.06298828125" disk.nvme0c0n1.in‚ 127.30078125 disk.nvme0n1.in‚ 127.30078125& disk.nvme0n1.out‚35031.97509765625$ memory_percent‚98.81489276885986 proc.memory.rssMB‚ 2871.59375 cpu‚26.457239713882416% disk./.usageGB‚1172.7935676574707 disk.dm-1.in‚0 disk.dm-1.out‚4082.595703125Í®<‚(: ÜËõÒÂÌ¥À gpu.0.gpu‚69.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚189.703' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚63.23433333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹`O=w‚(:ì ëËõÒ˜¾ö¿ network.sent‚ 1.294219e+07 network.recv‚ 1.139494e+06$ disk./.usageGB‚1172.918643951416 disk.dm-1.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚35057.80908203125 disk.dm-0.in‚ 139.4140625 disk.rbd0.out‚0! disk.nvme1c1n1.in‚ 12.11328125( proc.memory.availableMB‚ 194.37890625 cpu‚26.455527064081927 disk.dm-1.out‚4082.662109375 disk.nvme0n1.in‚ 127.30078125 disk.nvme1n1.in‚ 12.11328125# disk.nvme1n1.out‚12487.90234375 disk.md0.in‚0$ memory_percent‚98.81360530853271) disk./.usagePercent‚76.39919517677356" disk.nvme0c0n1.in‚ 127.30078125% disk.nvme1c1n1.out‚12487.90234375 disk.md0.out‚ 134.08984375# disk.dm-0.out‚43178.51416015625 proc.memory.rssMB‚ 2871.59375* proc.memory.percent‚17.526817321777344 proc.cpu.threads‚17& disk.nvme0n1.out‚35057.80908203125'4š*›Ž‚(: ëËõÒñá¿ gpu.0.gpu‚10.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚178.911' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚59.63699999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë…¼J2- _wandbruntime‚3990  _runtime‚3990<˜ÉIÇÁ _step‚788 & lr-AdamW/pg1‚9.835176113606449e-05 & lr-AdamW/pg2‚9.835176113606449e-05 trainer/global_step‚9749 % _timestamp‚1.7845058365426798e+09 _runtime‚3990.605831898ïÌ‘Æ & lr-AdamW/pg1‚9.835176113606449e-05 & lr-AdamW/pg2‚9.835176113606449e-05 trainer/global_step‚9749 % _timestamp‚1.7845058365426798e+09 _runtime‚3990.605831898 _step‚788”4­O¬2’- _wandbruntime‚3990  _runtime‚3990FÙG¯Ø'“ÒO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚376003461878784 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 trainer/global_step‚9749 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6841749096660 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74233353 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚376003444839424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74233355 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74233355 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382866621007872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173114407 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6863264190464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚366214047208960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7064471199232 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 % _timestamp‚1.7845058365434928e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚382866726069248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6863176168448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98881052 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚366213843988992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98870909 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7064649515520 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42958647 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42958644 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6863176168448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚371133780758576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 Á ”×O WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6863176168448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98881052 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚366213843988992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98870909 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7064649515520 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42958647 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42958644 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6863176168448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚371133780758576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚376003461878784 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 trainer/global_step‚9749 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6841749096660 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74233353 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚376003444839424 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74233355 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74233355 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚382866621007872 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173114407 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6863264190464 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚366214047208960 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7064471199232 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 % _timestamp‚1.7845058365434928e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚382866726069248•gE¤2•- _wandbruntime‚3991  _runtime‚3991%»àâ&–ÜM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚359185092850688 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚376040828246016 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 trainer/global_step‚9749 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚376040828246016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173121481 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25067863 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98880807 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74240674 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚378013077874440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7845058371027424e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173131640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68030800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42962937 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚376040811206656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚366250444359680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚359185067947008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98880807 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98890964 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 _step‚790 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚371170669865696 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _runtime‚3991.166055638 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173131640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25067618 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6863942307328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68030552 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7065169005568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74240676 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚382904770553344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6863854278144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚371170652826336 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74240676 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6863854278144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚382904770553344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6863942307328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6842511196690 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚371170652826336 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74240676 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6863854278144 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚382904770553344 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6863942307328 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6842511196690 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚376040828246016 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 trainer/global_step‚9749 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚376040828246016 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173121481 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25067863 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98880807 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74240674 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚378013077874440 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7845058371027424e+09 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173131640 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68030800 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚42962937 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚376040811206656 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚366250444359680 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚359185067947008 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚98880807 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98890964 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 _step‚790 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚371170669865696 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _runtime‚3991.166055638 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173131640 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25067618 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6863942307328 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68030552 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7065169005568 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74240676 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚382904770553344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6863854278144 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0–“ Ú2˜- _wandbruntime‚3991  _runtime‚3991§Ó#K™ _runtime‚3991.166779939 1loss/SRqqt1/assignment_loss‚0.480339608513223 _step‚791 0loss/FRt1/assignment_loss‚2.1977493608922183 1loss/FBt2/assignment_loss‚0.00312018301437088 /loss/FRt1/detection_loss‚0.5137642583950018 /loss/FRt2/detection_loss‚1.4386071635294273 1loss/SRqqt1/detection_loss‚0.9766719848995313 /loss/FBt1/detection_loss‚1.0450915481247331 /loss/FBt2/detection_loss‚0.4167830763326492 epoch‚12 0loss/FRt2/assignment_loss‚0.8396822764126528 2loss/SRqqt2/assignment_loss‚0.5635623383925468 2loss/FBt1/assignment_loss‚0.004178342124325117 1loss/SRqqt2/detection_loss‚0.6672258932854415 %loss/total_loss‚9.146776033916122 trainer/global_step‚9749 $ _timestamp‚1.784505837103727e+09HÃ_· š† 1loss/SRqqt1/assignment_loss‚0.480339608513223 _step‚791 0loss/FRt1/assignment_loss‚2.1977493608922183 1loss/FBt2/assignment_loss‚0.00312018301437088 /loss/FRt1/detection_loss‚0.5137642583950018 /loss/FRt2/detection_loss‚1.4386071635294273 1loss/SRqqt1/detection_loss‚0.9766719848995313 /loss/FBt1/detection_loss‚1.0450915481247331 /loss/FBt2/detection_loss‚0.4167830763326492 epoch‚12 0loss/FRt2/assignment_loss‚0.8396822764126528 2loss/SRqqt2/assignment_loss‚0.5635623383925468 2loss/FBt1/assignment_loss‚0.004178342124325117 1loss/SRqqt2/detection_loss‚0.6672258932854415 %loss/total_loss‚9.146776033916122 trainer/global_step‚9749 $ _timestamp‚1.784505837103727e+09 _runtime‚3991.166779939—Ö†»z›‚(:ï úËõÒ¬ùì¿ disk.nvme1n1.in‚ 12.11328125 disk.md0.out‚ 134.3671875 disk.dm-0.in‚ 139.4140625 disk.rbd0.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1172.9185104370117 disk.nvme0n1.in‚ 127.30078125! disk.nvme1c1n1.in‚ 12.11328125( disk.nvme0c0n1.out‚35059.63037109375$ memory_percent‚99.20840263366699' proc.memory.availableMB‚ 129.6953125* proc.memory.percent‚17.526817321777344 disk.md0.in‚0# disk.dm-0.out‚43180.17626953125" disk.nvme0c0n1.in‚ 127.30078125 proc.memory.rssMB‚ 2871.59375 network.sent‚ 1.2945112e+07 network.recv‚ 1.142828e+06& disk.nvme0n1.out‚35059.63037109375% disk.nvme1n1.out‚12488.5673828125 disk.dm-1.in‚0 disk.dm-1.out‚ 4082.86328125 disk.rbd0.in‚0 cpu‚26.453890164533963) disk./.usagePercent‚76.39918648018315' disk.nvme1c1n1.out‚12488.5673828125iE§ïœ‚(:„ úËõÒꇊÀ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚112.035' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.345 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨·ü¥2- _wandbruntime‚4008  _runtime‚4008`—‡UÅž¿ % _timestamp‚1.7845058546350808e+09 _runtime‚4008.700604348 _step‚792 % lr-AdamW/pg1‚9.82041128807919e-05 % lr-AdamW/pg2‚9.82041128807919e-05 trainer/global_step‚9799 à)dÊŸÄ trainer/global_step‚9799 % _timestamp‚1.7845058546350808e+09 _runtime‚4008.700604348 _step‚792 % lr-AdamW/pg1‚9.82041128807919e-05 % lr-AdamW/pg2‚9.82041128807919e-05˜ì!©2 - _wandbruntime‚4008  _runtime‚4008ƃõ¡ÑO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚360934155555840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚377871780240384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚360934130652160 GADeviceStats‘9[tH&Monitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173967064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 _runtime‚4008.700660978 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74599405 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25188772 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74599405 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68361663 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784505854635617e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚384768982172672 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚384768877111296 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚372977401185936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚377871780240384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚377871763201024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7099587090944 _step‚793 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99367661 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99367661 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6897113910272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚384768877111296 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6897113910272 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173977209 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43173147 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68361919 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚9799 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚368033547815424 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6875664720542 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99377804 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚377871763201024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169927680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99377804 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74599403 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚379852962718644 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6897201932288 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74599403 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194831360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7099417163264 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25188519 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚372977384146576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43173144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330ð {Ü'¢ÖO TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74599405 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25188772 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74599405 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68361663 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.784505854635617e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚384768982172672 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚384768877111296 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚372977401185936 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚377871780240384 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚377871763201024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7099587090944 _step‚793 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99367661 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99367661 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6897113910272 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚384768877111296 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6897113910272 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173977209 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43173147 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68361919 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚9799 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚368033547815424 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6875664720542 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99377804 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚377871763201024 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169927680 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99377804 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74599403 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚379852962718644 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6897201932288 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74599403 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194831360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7099417163264 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25188519 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚372977384146576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43173144 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚360934155555840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚377871780240384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚360934130652160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173967064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 _runtime‚4008.700660978™˜V˜ã2£- _wandbruntime‚4008  _runtime‚4008éû`á&¤ÛM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚377909146607616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43177434 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173994442 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚360969825809408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6897792019968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74606724 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚377909146607616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚368069938198528 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25190900 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚373014273253696 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚377909129568256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚373014256214336 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 trainer/global_step‚9799 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173994442 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚360969850713088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚384807026656768 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚384806921588224 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99387716 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43177437 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6897792019968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7100112389120 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚379890510737848 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 % _timestamp‚1.7845058548844965e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68368334 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚379890613925794 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74606724 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99387716 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6897880049152 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚368070143508480 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6876340672098 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173984283 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6897880049152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68368577 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 _runtime‚ 4008.94777745 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74606726 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74606726 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25191140 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7100292795392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚377909129568256ß¾Ŧ ¥àM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚377909146607616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚368069938198528 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25190900 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚373014273253696 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚377909129568256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚373014256214336 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 trainer/global_step‚9799 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 Bþ@rain_batch_end/inactive_split.all.current‚243 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 173984283 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173994442 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚360969850713088 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚384807026656768 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚384806921588224 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99387716 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43177437 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6897792019968 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7100112389120 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚379890510737848 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 % _timestamp‚1.7845058548844965e+09 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68368334 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚379890613925794 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74606724 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99387716 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6897880049152 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚368070143508480 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6876340672098 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173984283 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6897880049152 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68368577 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 _runtime‚ 4008.94777745 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74606726 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74606726 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25191140 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7100292795392 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚377909129568256 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚377909146607616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43177434 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173994442 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚360969825809408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6897792019968 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74606724šÏ3<2¦- _wandbruntime‚4008  _runtime‚4008prB§€ 1loss/FBt1/assignment_loss‚0.07569616927436713 2loss/SRqqt1/assignment_loss‚0.6212888672684547 1loss/SRqqt2/assignment_loss‚0.587426837791947 /loss/FBt2/detection_loss‚0.5267365404288284 _runtime‚4008.948220431 2loss/FBt2/assignment_loss‚0.001731750388344544 /loss/FRt1/detection_loss‚0.5651233919451037 /loss/FRt2/detection_loss‚1.4809159613115357 0loss/SRqqt1/detection_loss‚0.974998184894941 1loss/SRqqt2/detection_loss‚0.5970146585992608 /loss/FBt1/detection_loss‚0.9386094564741309 %loss/total_loss‚9.368484264032338 % _timestamp‚1.7845058548853562e+09 /loss/FRt1/assignment_loss‚2.149642144580443 0loss/FRt2/assignment_loss‚0.8493003010749817 epoch‚12 trainer/global_step‚9799 _step‚795§< ¨… trainer/global_step‚9799 _step‚795 1loss/FBt1/assignment_loss‚0.07569616927436713 2loss/SRqqt1/assignment_loss‚0.6212888672684547 1loss/SRqqt2/assignment_loss‚0.587426837791947 /loss/FBt2/detection_loss‚0.5267365404288284 _runtime‚4008.948220431 2loss/FBt2/assignment_loss‚0.001731750388344544 /loss/FRt1/detection_loss‚0.5651233919451037 /loss/FRt2/detection_loss‚1.4809159613115357 0loss/SRqqt1/detection_loss‚0.974998184894941 1loss/SRqqt2/detection_loss‚0.5970146585992608 /loss/FBt1/detection_loss‚0.9386094564741309 %loss/total_loss‚9.368484264032338 % _timestamp‚1.7845058548853562e+09 /loss/FRt1/assignment_loss‚2.149642144580443 0loss/FRt2/assignment_loss‚0.8493003010749817 epoch‚12›-àÚ›©‚(: ‰ÌõÒÌ׃À gpu.0.gpu‚62.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚177.062' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚59.02066666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÉkÅÐ|ª‚(:ñ ‰ÌõÒ÷‰¤À) disk./.usagePercent‚76.41469672494556% disk./.usageGB‚1173.1566314697266 disk.dm-1.out‚ 4083.02734375 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme1c1n1.in‚ 12.11328125' disk.nvme1c1n1.out‚12575.4013671875$ memory_percent‚99.21033382415771 network.sent‚ 1.3007637e+07 network.recv‚ 1.146872e+06" disk.nvme0c0n1.in‚ 127.30078125& disk.nvme0n1.out‚35229.02490234375 disk.nvme1n1.in‚ 12.11328125( proc.memory.availableMB‚ 129.37890625 proc.memory.rssMB‚ 2871.59375* proc.memory.percent‚17.526817321777344 disk.nvme0n1.in‚ 127.30078125% disk.nvme1n1.out‚12575.4013671875 cpu‚26.452258605445053 disk.dm-0.in‚ 139.4140625# disk.dm-0.out‚43435.52587890625 disk.dm-1.in‚0( disk.nvme0c0n1.out‚35229.02490234375 disk.md0.in‚0 disk.md0.out‚ 134.69921875 proc.cpu.threads‚17É5­x2«- _wandbruntime‚4027  _runtime‚4027ªá—ǬÁ & lr-AdamW/pg1‚9.805578965819985e-05 & lr-AdamW/pg2‚9.805578965819985e-05 trainer/global_step‚9849 % _timestamp‚1.7845058735340576e+09 _runtime‚4027.597277403 _step‚796RX™3Ì­Æ trainer/global_step‚9849 % _timestamp‚1.7845058735340576e+09 _runtime‚4027.597277403 _step‚796 & lr-AdamW/pg1‚9.805578965819985e-05 & lr-AdamW/pg2‚9.805578965819985e-05œU¬§2®- _wandbruntime‚4027  _runtime‚4027†`Œ×'¯ÑO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7134408221184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43387647 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚381730498769886 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚379740081562624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74965453 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚362718913418240 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚374820987534576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6909494195950 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚386671238276096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25312708 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚362718888514560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6931139674112 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68700117 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚797 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68700355 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 174829866 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74965455 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6931051652096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚379740081562624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚381730395582052 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚4027.598530085 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386671133214720 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚379740098601984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚379740098601984 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99864413 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚369853296735744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7134582343168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74965453 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6909408047476 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74965455 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 $ _timestamp‚1.784505873535013e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚03€zîÜ'°ÖO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74965453 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚362718913418240 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚374820987534576 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6909494195950 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚386671238276096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25312708 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚362718888514560 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6931139674112 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚68700117 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚797 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68700355 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 174829866 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74965455 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6931051652096 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚379740081562624 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚381730395582052 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚4027.598530085 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚386671133214720 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚379740098601984 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚379740098601984 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚99864413 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚369853296735744 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7134582343168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74965453 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6909408047476 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74965455 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 $ _timestamp‚1.784505873535013e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7134408221184 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚43387647 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚381730498769886 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚379740081562624 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0ÍÜ4„2±- _wandbruntime‚4027  _runtime‚4027» %(²ÜM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43391934 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚381767943601256 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚379777464969216 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99874311 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43391937 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 aODeviceStatsMonitor.on_train_batch_end/inkl+ëº active_split_bytes.small_pool.allocated‚ 7135289573376 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74972774 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚379777447929856 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _runtime‚4027.856046641 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 174857244 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚369889684450304 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6931817790976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 174847085 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99884468 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚381768046789202 _step‚798 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74972776 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6931729761792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25314965 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74972774 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚379777447929856 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚99874311 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚374857859602336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚386709282760192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚386709177691648 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚273 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚270 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68706899 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚362754608575488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6910170147506 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚374857876641696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚362754583671808 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 174857244 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚386709282760192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 174847085 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99884468 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 % _timestamp‚1.7845058737927701e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25315235 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6931817790976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68706899 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚362754608575488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6910170147506 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚374857876641696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚362754583671808 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 174857244 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚386709282760192 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 174847085 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99884468 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 % _timestamp‚1.7845058737927701e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25315235 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6931817790976 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74972774 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚379777447929856 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _runtime‚4027.856046641 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 174857244 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚369889684450304 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6931817790976 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 174847085 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99884468 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚381768046789202 _step‚798 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74972776 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6931729761792 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25314965 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0žQEŸ[2´- _wandbruntime‚4027  _runtime‚40278éS µƒ 1loss/SRqqt2/detection_loss‚0.6362226597289622 /loss/FBt1/detection_loss‚0.8348646075982187 0loss/FRt2/assignment_loss‚0.8831629760323032 2loss/SRqqt1/assignment_loss‚0.5803322869708621 2loss/FBt1/assignment_loss‚0.004009883383888437 3loss/FBt2/assignment_loss‚0.0015392842622907249 0loss/FRt1/assignment_loss‚2.1574841228598687 epoch‚12 trainer/global_step‚9849 /loss/FBt2/detection_loss‚0.6564373431740225 %loss/total_loss‚9.452884395330841 1loss/SRqqt2/assignment_loss‚0.613197014754989 /loss/FRt1/detection_loss‚0.5804125400773607 .loss/FRt2/detection_loss‚1.434995677205734 1loss/SRqqt1/detection_loss‚1.0702259992823413 % _timestamp‚1.7845058737937014e+09 _runtime‚4027.856705372 _step‚799©·¶ˆ 1loss/SRqqt2/detection_loss‚0.6362226597289622 /loss/FBt1/detection_loss‚0.8348646075982187 0loss/FRt2/assignment_loss‚0.8831629760323032 2loss/SRqqt1/assignment_loss‚0.5803322869708621 2loss/FBt1/assignment_loss‚0.004009883383888437 3loss/FBt2/assignment_loss‚0.0015392842622907249 0loss/FRt1/assignment_loss‚2.1574841228598687 epoch‚12 trainer/global_step‚9849 /loss/FBt2/detection_loss‚0.6564373431740225 %loss/total_loss‚9.452884395330841 1loss/SRqqt2/assignment_loss‚0.613197014754989 /loss/FRt1/detection_loss‚0.5804125400773607 .loss/FRt2/detection_loss‚1.434995677205734 1loss/SRqqt1/detection_loss‚1.0702259992823413 % _timestamp‚1.7845058737937014e+09 _runtime‚4027.856705372 _step‚799ŸuŠ@v·‚(:ë ˜ÌõÒ¬Óú¿) disk./.usagePercent‚76.41470070052974' disk.nvme1c1n1.out‚12577.4091796875 disk.nvme1n1.in‚ 12.11328125% disk.nvme1n1.out‚12577.4091796875 disk.dm-0.in‚ 139.4140625$ memory_percent‚99.20902252197266% disk./.usageGB‚1173.1566925048828 disk.dm-1.in‚0 disk.nvme0n1.in‚ 127.30078125% disk.nvme0n1.out‚35237.9970703125! disk.nvme1c1n1.in‚ 12.11328125 disk.md0.out‚ 135.140625" disk.dm-0.out‚43445.3974609375 proc.memory.rssMB‚ 2871.59375! disk.dm-1.out‚4083.1767578125 disk.rbd0.in‚0 cpu‚26.450402274929694 proc.cpu.threads‚17 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 127.30078125' disk.nvme0c0n1.out‚35237.9970703125 disk.md0.in‚0% proc.memory.availableMB‚ 129.59375* proc.memory.percent‚17.526817321777344 network.sent‚ 1.3068983e+07 network.recv‚ 1.150663e+06i¨…ü¸‚(:… ˜ÌõÒ¹™ÚÀ gpu.0.gpu‚74.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚115.458' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚38.486 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CÔ—2¹- _wandbruntime‚4046  _runtime‚4046 2ÚõǺÁ & lr-AdamW/pg1‚9.790679384398581e-05 & lr-AdamW/pg2‚9.790679384398581e-05 trainer/global_step‚9899 % _timestamp‚1.7845058925336728e+09 _runtime‚4046.598598643 _step‚800fÀÓãÌ»Æ % _timestamp‚1.7845058925336728e+09 _runtime‚4046.598598643 _step‚800 & lr-AdamW/pg1‚9.790679384398581e-05 & lr-AdamW/pg2‚9.790679384398581e-05 trainer/global_step‚9899 ßH2¼- _wandbruntime‚4046  _runtime‚4046»‰ÑSÜ'½ÖO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6965077415936 _runtime‚4046.598660313 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75331505 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6964989393920 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75331505 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7169389882880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100371308 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6943237522884 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100371308 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25436111 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43602144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚388573494379520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6965077415936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75331503 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7169570296320 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 trainer/global_step‚9899 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚388573389318144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚381608399924224 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚376664607961936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6943323671358 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚383607931633294 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚383607828445460 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845058925343533e+09 _step‚801 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75331503 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚381608416963584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175702813 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25436368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚364503646376960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚381608416963584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175692668¢\¯â¾ÛO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceSºì¥±ÿ$tatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚383607931633294 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚383607828445460 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845058925343533e+09 _step‚801 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75331503 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚381608416963584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175702813 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25436368 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚364503646376960 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚381608416963584 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175692668 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6965077415936 _runtime‚4046.598660313 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚75331505 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6964989393920 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75331505 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7169389882880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100371308 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6943237522884 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100371308 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25436111 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43602144 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚388573494379520 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6965077415936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75331503 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7169570296320 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 trainer/global_step‚9899 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚388573389318144 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚381608399924224 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚376664607961936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6943323671358 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚383645479652610 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6965755532800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7170269492224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚383645376464664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69045013 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175720046 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚371709432717312 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25438827 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚381645766291456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚381645783330816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75338826 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69045264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6965667503616 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6943913474328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚381645783330816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚376701462990336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175709887 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚388611538863616 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚4046.856122206 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6965667503616 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚371709635930112 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7170091183104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6943999622914 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43606437 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25438579 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175709887 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175720046 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚371709432717312 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25438827 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚381645766291456 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚381645783330816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75338826 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69045264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6965667503616 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6943913474328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚381645783330816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚376701462990336 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 175709887 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚388611538863616 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚4046.856122206 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6965667503616 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚371709635930112 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7170091183104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6943999622914 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43606437 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25438579 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175709887 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚383645479652610 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6965755532800 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7170269492224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚383645376464664 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69045013 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚385485261308868 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚383476718285824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 _runtime‚4065.263530736 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75697553 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚378508211349936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6999015157760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25559902 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚366288404239360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚390475750482944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7204523778560 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚383476735325184 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75697555 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚383476718285824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69376549 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43816644 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚373492751798784 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 trainer/global_step‚9949 _step‚805 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚383476735325184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100868060 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6998927135744 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚390475750482944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100868060 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75697553 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚390475645421568 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100857917 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448nìÊêá'ÎÛO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚383476718285824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 _runtime‚4065.263530736 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚75697553 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚378508211349936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6999015157760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25559902 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚366288404239360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚390475750482944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7204523778560 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚383476735325184 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75697555 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 176219136 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚383476718285824 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69376549 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚43816644 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚373492751798784 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 trainer/global_step‚9949 _step‚805 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚383476735325184 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100868060 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6998927135744 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚390475750482944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100868060 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚75697553 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚390475645421568 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100857917 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6999015157760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7204347559424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚378508194310576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 176555470 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚385485261308868¥ÕiŸ2Ï- _wandbruntime‚4065  _runtime‚4065Ê‹æ&ÐàM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6977742949736 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚385522912516018 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6977829098322 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75704876 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100867815 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69383003 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _step‚806 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176582848 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚385522809328072 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚378545066378336 trainer/global_step‚9949 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100877972 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6999605245440 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25562069 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7205228762112 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6999693274624 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100867815 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚383514101692416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚366324124300288 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚383514101692416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚390513794967040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75704874 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 176572689 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43820934 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7205046258688 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75704876 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6999693274624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚378545083417696 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚390513689898496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25562336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75704874 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6999605245440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43820937 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚383514084653056 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚366324099396608 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 % _timestamp‚1.7845059117772574e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚390513794967040 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚383514101692416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚390513794967040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75704874 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 176572689 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43820934 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7205046258688 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75704876 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6999693274624 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚378545083417696 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚390513689898496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25562336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75704874 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6999605245440 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43820937 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚383514084653056 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚366324099396608 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 % _timestamp‚1.7845059117772574e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚390513794967040 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69383003 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _step‚806 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176582848 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚385522809328072 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚378545066378336 trainer/global_step‚9949 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100877972 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6999605245440 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚267 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0¦*¸šy2Ò- _wandbruntime‚4065  _runtime‚4065#Û™Ó‚ 2loss/FBt2/assignment_loss‚0.008117068774186414 1loss/SRqqt1/detection_loss‚1.0098931263930289 epoch‚12 trainer/global_step‚9949 _step‚807 2loss/SRqqt1/assignment_loss‚0.6056786540054845 2loss/FBt1/assignment_loss‚0.004135367168027908 /loss/FBt2/detection_loss‚0.6046427892521024 _runtime‚4065.843573703 0loss/FRt2/assignment_loss‚0.9823161805829694 .loss/FRt2/detection_loss‚1.512391971449973 /loss/FBt1/detection_loss‚0.5686961989073703 %loss/total_loss‚9.305977294036508 /loss/FRt1/detection_loss‚0.5277014824088416 1loss/SRqqt2/detection_loss‚0.6302542668916951 % _timestamp‚1.7845059117782888e+09 /loss/FRt1/assignment_loss‚2.228770844474762 2loss/SRqqt2/assignment_loss‚0.6233793437280665ñŒ Ô‡ /loss/FRt1/detection_loss‚0.5277014824088416 1loss/SRqqt2/detection_loss‚0.6302542668916951 % _timestamp‚1.7845059117782888e+09 /loss/FRt1/assignment_loss‚2.228770844474762 2loss/SRqqt2/assignment_loss‚0.6233793437280665 2loss/FBt2/assignment_loss‚0.008117068774186414 1loss/SRqqt1/detection_loss‚1.0098931263930289 epoch‚12 trainer/global_step‚9949 _step‚807 2loss/SRqqt1/assignment_loss‚0.6056786540054845 2loss/FBt1/assignment_loss‚0.004135367168027908 /loss/FBt2/detection_loss‚0.6046427892521024 _runtime‚4065.843573703 0loss/FRt2/assignment_loss‚0.9823161805829694 .loss/FRt2/detection_loss‚1.512391971449973 /loss/FBt1/detection_loss‚0.5686961989073703 %loss/total_loss‚9.305977294036508§[öMÕ‚(:ô ÅÌõÒñÕç¿ network.sent‚ 1.3198538e+07( disk./.usagePercent‚76.4147588434485( disk.nvme1c1n1.out‚12582.03271484375* proc.memory.percent‚17.526817321777344 network.recv‚ 1.162402e+06$ disk./.usageGB‚1173.157585144043 disk.nvme1n1.in‚ 12.11328125& disk.nvme1n1.out‚12582.03271484375( disk.nvme0c0n1.out‚35285.67919921875& disk.nvme0n1.out‚35285.67919921875! disk.nvme1c1n1.in‚ 12.11328125$ memory_percent‚99.20871257781982 disk.md0.in‚0 disk.dm-0.in‚ 139.4140625# disk.dm-0.out‚43493.42626953125 disk.rbd0.out‚0 disk.nvme0n1.in‚ 127.30078125( proc.memory.availableMB‚ 129.64453125 proc.cpu.threads‚17 disk.md0.out‚ 136.16796875 disk.dm-1.in‚0" disk.dm-1.out‚4085.24853515625 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 127.30078125 proc.memory.rssMB‚ 2871.59375 cpu‚26.444843013313374è&Ö]œÖ‚(:‘ ÅÌõÒ¯ª¿À gpu.0.gpu‚83.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚112.427' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.47566666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X ù2×- _wandbruntime‚4084  _runtime‚4084½iË]Ãؽ trainer/global_step‚9999 % _timestamp‚1.7845059302070882e+09 _runtime‚4084.270263639 _step‚808 $ lr-AdamW/pg1‚9.7606793997308e-05 $ lr-AdamW/pg2‚9.7606793997308e-05.É`ÈÙ _runtime‚4084.270263639 _step‚808 $ lr-AdamW/pg1‚9.7606793997308e-05 $ lr-AdamW/pg2‚9.7606793997308e-05 trainer/global_step‚9999 % _timestamp‚1.7845059302070882e+09¨ÚÿQ‡2Ú- _wandbruntime‚4084  _runtime‚4084]IÒÜ'ÛÖO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚392378006586368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚385345053686784 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7010896473700 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7032952899584 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69714599 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _step‚809 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚392377901524992 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101364812 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚375312515196928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7239543994368 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 trainer/global_step‚9999 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177428417 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25683719 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101354669 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚392378006586368 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7010982622174 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚385345053686784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44031144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76063605 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚375312730999808 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚385345036647424 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69714866 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44031147 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76063603 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76063605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚392377901524992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76063603 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215802880 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7032864877568 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101354669 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0*ÑÁ1Ñ&ÜÛO VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚385345036647424 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69714866 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44031147 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76063603 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76063605 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚392377901524992 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76063603 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215802880 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7032864877568 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101354669 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚392378006586368 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚385345053686784 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7010896473700 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7032952899584 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69714599 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _step‚809 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚392377901524992 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101364812 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚375312515196928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7239543994368 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 trainer/global_step‚9999 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 177428417 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25683719 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101354669 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚392378006586368 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7010982622174 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚385345053686784 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44031144 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76063605 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚œh0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚375312730999808©!>;2Ý- _wandbruntime‚4084  _runtime‚4084ÇvCå&ÞßM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7033542987264 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚385382420054016 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7033542987264 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76070924 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69721412 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101374724 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7033631016448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25685978 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7240235184640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177445650 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174114816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7240061069824 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 _runtime‚ 4084.52832661 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7033631016448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚380388669766336 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76070924 % _timestamp‚1.7845059304650207e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚392416051070464 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199018496 trainer/global_step‚9999 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚385382420054016 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69721681 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚368108882162688 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚392415946001920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25686244 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177445650 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44035437 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚387400242191480 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101364567 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚385382403014656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76070926 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7011658573730 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚385382403014656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚368108857259008 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚387400345379426 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76070926 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚375348918328832 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44035434 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚375349117347328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816©}"Xê&ßäM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 177435491 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚392416051070464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69721412 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 101374724 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7033631016448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25685978 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7240235184640 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 177445650 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 174114816 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7240061069824 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 _runtime‚ 4084.52832661 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7033631016448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚380388669766336 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76070924 % _timestamp‚1.7845059304650207e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚392416051070464 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 199018496 trainer/global_step‚9999 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚385382420054016 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69721681 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚368108882162688 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚392415946001920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25686244 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 177445650 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44035437 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚387400242191480 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101364567 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚385382403014656 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76070926 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7011658573730 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚385382403014656 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚368108857259008 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚387400345379426 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76070926 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚375348918328832 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44035434 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚375349117347328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7033542987264 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚385382420054016 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7033542987264 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76070924 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚394280157628416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76429653 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚377132303267840 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25807395 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚369857919964160 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚387213355009024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178291219 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7066802619392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 _step‚813 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70053297 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7274572105728 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚394280262689792 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178281074 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚394280157628416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚387213372048384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25807650 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7066890641408 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7044812097582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101851421 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7066890641408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7044725949108 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845059492024968e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7066802619392 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7066890641408 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7044812097582 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101851421 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7066890641408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚258 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7044725949108 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7845059492024968e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7066802619392 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚394280157628416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76429653 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚377132303267840 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25807395 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚369857919964160 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚387213355009024 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178291219 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7066802619392 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 _step‚813 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70053297 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7274572105728 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚394280262689792 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178281074 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚394280157628416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚387213372048384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25807650 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104­¶ Yþ2ë- _wandbruntime‚4103  _runtime‚4103/رäç&ìáM E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7067480729088 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7067568758272 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178308452 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚387250738415616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25809857 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚389277778242834 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76436976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7275087454720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚387250721376256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70059791 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101871476 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44249937 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚394318307173888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7845059494608924e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚394318307173888 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7275265763840 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚387250738415616 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚377168702576128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _step‚814 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚10049 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76436974 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚394318202105344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚387250721376256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚389277675054888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178298293 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76436976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101861319 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚377168905788928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178308452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚382232273154336>T–¶ì&íæM OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 101861319 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚377168905788928 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178308452 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚382232273154336 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7067480729088 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7067568758272 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178308452 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚387250738415616 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25809857 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚389277778242834 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76436976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7275087454720 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚387250721376256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70059791 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101871476 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44249937 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚394318307173888 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7845059494608924e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚394318307173888 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7275265763840 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚387250738415616 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚377168702576128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _step‚814 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚10049 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76436974 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚394318202105344 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚387250721376256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚389277675054888 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178298293 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76436976 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816®*•ò!2î- _wandbruntime‚4103  _runtime‚4103ˆdö™ ï… /loss/FBt2/detection_loss‚0.5769319443963468 trainer/global_step‚10049 _runtime‚4103.524671263 /loss/FRt1/assignment_loss‚2.089090108994865 2loss/FBt2/assignment_loss‚0.014742979407905412 /loss/FRt1/detection_loss‚0.5681512327705539 /loss/FBt1/detection_loss‚1.0795975161771125 epoch‚12 % _timestamp‚1.7845059494618378e+09 _step‚815 3loss/FBt1/assignment_loss‚0.0025440196105752395 %loss/total_loss‚9.912903606507246 2loss/SRqqt2/assignment_loss‚0.5727114008929182 1loss/SRqqt1/detection_loss‚1.1267867118691004 1loss/SRqqt2/detection_loss‚0.6071579721436969 0loss/FRt2/assignment_loss‚0.8322467450642478 2loss/SRqqt1/assignment_loss‚0.7656030098877606 /loss/FRt2/detection_loss‚1.6773399652921641uŠÝþðŠ 2loss/SRqqt1/assignment_loss‚0.7656030098877606 /loss/FRt2/detection_loss‚1.6773399652921641 /loss/FBt2/detection_loss‚0.5769319443963468 trainer/global_step‚10049 _runtime‚4103.524671263 /loss/FRt1/assignment_loss‚2.089090108994865 2loss/FBt2/assignment_loss‚0.014742979407905412 /loss/FRt1/detection_loss‚0.5681512327705539 /loss/FBt1/detection_loss‚1.0795975161771125 epoch‚12 % _timestamp‚1.7845059494618378e+09 _step‚815 3loss/FBt1/assignment_loŸvss‚0.0025440196105752395 %loss/total_loss‚9.912903606507246 2loss/SRqqt2/assignment_loss‚0.5727114008929182 1loss/SRqqt1/detection_loss‚1.1267867118691004 1loss/SRqqt2/detection_loss‚0.6071579721436969 0loss/FRt2/assignment_loss‚0.8322467450642478¯¹\Èñ‚(:ô ãÌõÒ™°Ó¿ proc.cpu.threads‚17 network.sent‚ 1.3324475e+07% disk./.usageGB‚1173.1579208374023 disk.nvme0n1.in‚ 127.30078125( disk.nvme1c1n1.out‚12583.96435546875& disk.nvme1n1.out‚12583.96435546875 disk.md0.out‚ 136.98046875# disk.dm-0.out‚43499.88134765625 disk.nvme1n1.in‚ 12.11328125 disk.md0.in‚0 disk.rbd0.in‚0$ memory_percent‚99.20930862426758& proc.memory.availableMB‚ 129.546875 proc.memory.rssMB‚ 2871.59375 cpu‚26.441245458351858) disk./.usagePercent‚76.41478070916152( disk.nvme0c0n1.out‚35292.31787109375& disk.nvme0n1.out‚35292.31787109375! disk.nvme1c1n1.in‚ 12.11328125* proc.memory.percent‚17.526817321777344 network.recv‚ 1.170838e+06" disk.nvme0c0n1.in‚ 127.30078125 disk.dm-0.in‚ 139.4140625 disk.dm-1.in‚0" disk.dm-1.out‚4085.62158203125 disk.rbd0.out‚0¶–á¬ò‚(:… ãÌõÒ‘ÎùÁ gpu.0.gpu‚55.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚203.079' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚67.693 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úÕ“2ó- _wandbruntime‚4122  _runtime‚4122RŽ‘”ÇôÁ trainer/global_step‚10099 $ _timestamp‚1.784505968322654e+09 _runtime‚4122.385840233 _step‚816 & lr-AdamW/pg1‚9.730413256110738e-05 & lr-AdamW/pg2‚9.730413256110738e-05c0±ÉÌõÆ trainer/global_step‚10099 $ _timestamp‚1.784505968322654e+09 _runtime‚4122.385840233 _step‚816 & lr-AdamW/pg1‚9.730413256110738e-05 & lr-AdamW/pg2‚9.730413256110738e-05°fL8Ü2ö- _wandbruntime‚4122  _runtime‚4122ÈQGÊÝ'÷×O PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76795705 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚391117663086926 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚817 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚396182413731840 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚384039021513936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚396182518793216 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76795703 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚389081673370624 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76795703 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚384039004474576 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102348173 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7309505028096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 % _timestamp‚1.7845059683237042e+09 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚391117559899092 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7100740361216 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378951994052608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚371642677826560 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7100740361216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚389081673370624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚396182413731840 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚378952207758336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7078555424516 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚389081690409984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7100828383232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25931266 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚371642702730240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102358316 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7078641572990 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚396182413731840 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚384039021513936 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚396182518793216 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76795703 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚389081673370624 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76795703 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚262 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚384039004474576 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102348173 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7309505028096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 % _timestamp‚1.7845059683237042e+09 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚391117559899092 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7100740361216 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚378951994052608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚371642677826560 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7100740361216 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚389081673370624 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚259 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚396182413731840 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚378952207758336 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7078555424516 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚389081690409984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7100828383232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25931266 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚371642702730240 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102358316 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7078641572990 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚389119039737856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44464437 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚378988598048256 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚391155211106242 _step‚818 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 % _timestamp‚1.7845059685941322e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚389119056777216 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚389119056777216 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102368228 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚384075876542336 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 179171254 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7079317524546 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76803024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7310200160768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102358071 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70397979 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚384075893581696 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76803024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25933545 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7079231375960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 179161095 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7101418470912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 trainer/global_step‚10099 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76803026 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102358071 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179171254 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25933801 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚378988388544000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 _runtime‚ 4122.65743161 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7079317524546 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76803024 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7310200160768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 102358071 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70397979 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚384075893581696 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76803024 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25933545 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7079231375960 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 179161095 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7101418470912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 trainer/global_step‚10099 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76803026 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102358071 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 179171254 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25933801 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚378988388544000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 _runtime‚ 4122.65743161 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚389119039737856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44464437 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚378988598048256 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚391155211106242 _step‚818 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 % _timestamp‚1.7845059685941322e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚389119056777216 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚389119056777216 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 102368228²y¸2ü- _wandbruntime‚4122  _runtime‚4122Qø>‡ý 0loss/SRqqt2/detection_loss‚0.651246285039269 /loss/FBt1/detection_loss‚0.9971317827235907 $loss/total_loss‚9.40521579559094 trainer/global_step‚10099 _step‚819 2loss/SRqqt2/assignment_loss‚0.5593099126830163 2loss/FBt1/assignment_loss‚0.003759303804872616 3loss/FBt2/assignment_loss‚0.0021675809911382007 .loss/FRt1/detection_loss‚0.553156203531727 1loss/SRqqt1/detection_loss‚0.9744646212600759 /loss/FBt2/detection_loss‚0.3176265383983946 % _timestamp‚1.7845059685958536e+09 _runtime‚4122.658729792 /loss/FRt1/assignment_loss‚2.316584909548525 1loss/SRqqt1/assignment_loss‚0.691359417716644 epoch‚12 0loss/FRt2/assignment_loss‚0.9260866032353825 /loss/FRt2/detection_loss‚1.4123226366583046ÞgcÒ þ† 3loss/FBt2/assignment_loss‚0.0021675809911382007 .loss/FRt1/detection_loss‚0.553156203531727 1loss/SRqqt1/detection_loss‚0.9744646212600759 /loss/FBt2/detection_loss‚0.3176265383983946 % _timestamp‚1.7845059685958536e+09 _runtime‚4122.658729792 /loss/FRt1/assignment_loss‚2.316584909548525 1loss/SRqqt1/assignment_loss‚0.691359417716644 epoch‚12 0loss/FRt2/assignment_loss‚0.9260866032353825 /loss/FRt2/detection_loss‚1.4123226366583046 0loss/SRqqt2/detection_loss‚0.651246285039269 /loss/FBt1/detection_loss‚0.9971317827235907 $loss/total_loss‚9.40521579559094 trainer/global_step‚10099 _step‚819 2loss/SRqqt2/assignment_loss‚0.5593099126830163 2loss/FBt1/assignment_loss‚0.003759303804872616³´¡«€xÿ‚(:í òÌõÒ×ÄŽÀ disk.dm-1.out‚4085.689453125 proc.cpu.threads‚17) disk./.usagePercent‚76.41474468042982( disk.nvme0c0n1.out‚35303.18017578125! disk.nvme1c1n1.in‚ 12.11328125 disk.nvme1n1.in‚ 12.11328125 disk.dm-0.in‚ 139.4140625& proc.memory.availableMB‚ 129.390625 proc.memory.rssMB‚ 2871.59375* proc.memory.percent‚17.526817321777344 network.sent‚ 1.3399792e+07 network.recv‚ 1.1754e+06 disk.md0.in‚0 disk.md0.out‚ 137.4765625 cpu‚26.439519408410263 disk.rbd0.in‚0 disk.rbd0.out‚0% disk.nvme1n1.out‚12585.0283203125 disk.dm-1.in‚0$ memory_percent‚99.21026229858398% disk./.usageGB‚1173.1573677062988" disk.nvme0c0n1.in‚ 127.30078125 disk.nvme0n1.in‚ 127.30078125& disk.nvme0n1.out‚35303.18017578125' disk.nvme1c1n1.out‚12585.0283203125# disk.dm-0.out‚43510.67724609375B䵑€‚(:† òÌõÒ¶òºÀ gpu.0.gpu‚89.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚187.899' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚62.633 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½zÊs‚(:è ÍõÒá¦ì¿& proc.memory.availableMB‚ 129.390625 network.sent‚ 1.344824e+07 network.recv‚ 1.179603e+06) disk./.usagePercent‚76.41474592279988" disk.nvme0c0n1.in‚ 127.30078125! disk.nvme1c1n1.in‚ 12.11328125 disk.md0.out‚ 137.70703125$ memory_percent‚99.21026229858398 proc.memory.rssMB‚ 2871.59375 disk.dm-1.in‚0 disk.rbd0.in‚0# disk.nvme0n1.out‚35324.42578125% disk.nvme1n1.out‚12585.5068359375 disk.md0.in‚0 cpu‚26.437574806359844 proc.cpu.threads‚17% disk./.usageGB‚1173.1573867797852 disk.dm-1.out‚4085.787109375 disk.nvme0n1.in‚ 127.30078125 disk.nvme1n1.in‚ 12.11328125* proc.memory.percent‚17.526817321777344 disk.rbd0.out‚0% disk.nvme0c0n1.out‚35324.42578125' disk.nvme1c1n1.out‚12585.5068359375 disk.dm-0.in‚ 139.4140625" disk.dm-0.out‚43531.8076171875ÃW¹pœ‚‚(:‘ ÍõÒÄ°¨À gpu.0.gpu‚88.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚203.335' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.77833333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_èDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7134678103040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚373427460592640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚373427435688960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7112384899924 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚398084774896640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7134766125056 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _step‚821 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70729106 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚392995095950334 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚398084669835264 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚390949991732224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚246 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚398084774896640 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77161755 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180006678 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180016823 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44674647 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102855068 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77161753 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102844925 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44674644 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7134678103040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4142.316661081 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚390950008771584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚392994992762500 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 180016823 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44674647 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 102855068 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77161753 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102844925 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44674644 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7134678103040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4142.316661081 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚390950008771584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚392994992762500 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7ŠÅ´¥134678103040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚373427460592640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚373427435688960 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7112384899924 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚398084774896640 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7134766125056 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 _step‚821 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚70729106 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚392995095950334 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚398084669835264 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚390949991732224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0µ&Ý 2‰- _wandbruntime‚4142  _runtime‚4142,Ÿrç&ŠáM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77169074 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚373463155749888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77169076 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70736020 trainer/global_step‚10149 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180023897 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚398122819380736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚393032540781704 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚4142.587023405 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102854823 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180034056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26056839 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚390987375138816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26057083 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7135356212736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚373463130846208 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7345170344448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102864980 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70735773 _step‚822ÓåE.ì&‹æM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7135356212736 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚373463130846208 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7345170344448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102864980 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70735773 _step‚822 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77169074 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚373463155749888 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77169076 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70736020 trainer/global_step‚10149 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180023897 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚398122819380736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚393032540781704 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚4142.587023405 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 102854823 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180034056 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26056839 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚390987375138816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26057083 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816¶ºD±Õ2Œ- _wandbruntime‚4142  _runtime‚4142œÞWt ƒ 2loss/FBt1/assignment_loss‚0.013441336775928386 /loss/FBt1/detection_loss‚1.1279687231504603 /loss/FBt2/detection_loss‚0.5744360111073605 &loss/total_loss‚10.329093051172714 trainer/global_step‚10149 % _timestamp‚1.7845059885247507e+09 _step‚823 2loss/SRqqt1/assignment_loss‚0.7406463024150858 /loss/FRt1/detection_loss‚0.5977255626349615 /loss/FRt2/detection_loss‚1.4142661512247288 0loss/SRqqt1/detection_loss‚1.091479841459876 epoch‚12 _runtime‚4142.587590176 0loss/FRt2/assignment_loss‚1.0303323345879714 2loss/SRqqt2/assignment_loss‚0.6545849940625119 1loss/FBt2/assignment_loss‚0.19377818492162555 1loss/SRqqt2/detection_loss‚0.6999321176449433 /loss/FRt1/assignment_loss‚2.190501491187261DVsŽŽˆ epoch‚12 _runtime‚4142.587590176 0loss/FRt2/assignment_loss‚1.0303323345879714 2loss/SRqqt2/assignment_loss‚0.6545849940625119 1loss/FBt2/assignment_loss‚0.19377818492162555 1loss/SRqqt2/detection_loss‚0.6999321176449433 /loss/FRt1/assignment_loss‚2.190501491187261 2loss/FBt1/assignment_loss‚0.013441336775928386 /loss/FBt1/detection_loss‚1.1279687231504603 /loss/FBt2/detection_loss‚0.5744360111073605 &loss/total_loss‚10.329093051172714 trainer/global_step‚10149 % _timestamp‚1.7845059885247507e+09 _step‚823 2loss/SRqqt1/assignment_loss‚0.7406463024150858 /loss/FRt1/detection_loss‚0.5977255626349615 /loss/FRt2/detection_loss‚1.4142661512247288 0loss/SRqqt1/detection_loss‚1.091479841459876·@ ºw‚(:ì ÍõÒ™À¤À& disk.nvme0n1.out‚35327.63916015625' disk.nvme1c1n1.out‚12586.5244140625 disk.md0.out‚ 138.08203125& proc.memory.availableMB‚ 129.109375) disk./.usagePercent‚76.41474492890383 disk.dm-0.in‚141.375! disk.dm-1.out‚4085.9404296875 cpu‚26.435754728128266# disk.dm-0.out‚43535.06298828125 disk.nvme0n1.in‚ 129.26171875! disk.nvme1c1n1.in‚ 12.11328125% disk.nvme1n1.out‚12586.5244140625 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚35327.63916015625 network.recv‚ 1.183067e+06 disk.nvme1n1.in‚ 12.11328125 disk.md0.in‚0" disk.nvme0c0n1.in‚ 129.26171875$ memory_percent‚99.21197891235352 proc.memory.rssMB‚ 2871.59375* proc.memory.percent‚17.526817321777344 proc.cpu.threads‚17 network.sent‚ 1.3467003e+07$ disk./.usageGB‚1173.157371520996F»ª#œ‚(:‘ ÍõÒ«‹ÎÀ gpu.0.gpu‚69.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚199.259' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.41966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÊÓ2‘- _wandbruntime‚4161  _runtime‚4161×[¤.Ç’Á & lr-AdamW/pg1‚9.699882892625523e-05 & lr-AdamW/pg2‚9.699882892625523e-05 trainer/global_step‚10199 $ _timestamp‚1.784506007841433e+09 _runtime‚4161.904575878 _step‚824Q…4ÆÌ“Æ trainer/global_step‚10199 $ _timestamp‚1.784506007841433e+09 _runtime‚4161.904575878 _step‚824 & lr-AdamW/pg1‚9.699882892625523e-05 & lr-AdamW/pg2‚9.699882892625523e-05¸VŠ» 2”- _wandbruntime‚4161  _runtime‚4161‡»n¡•ÖO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26178235 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚375212218455040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚399986925938688 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77527805 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚382591448381952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 $ _timestamp‚1.784506007842384e+09 _runtime‚4161.905435859 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚399986925938688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26177985 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77527803 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103341677 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚392818327133184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 _step‚825 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 716861.g; 5844864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71067129 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44889147 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7146300523806 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103341677 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚375212193551360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚392818310093824 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚392818310093824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7168703866880 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103351820 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174121984 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚394872528813742 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚399987031000064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚387726211250576 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180869480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7168615844864 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚382591647407616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚394872425625908 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚392818327133184 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7146214375332 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208ÄÔ•zá'–ÛO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103351820 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77527803 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 $ _timestamp‚1.784506007842384e+09 _runtime‚4161.905435859 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚399986925938688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26177985 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77527803 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103341677 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚392818327133184 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 _step‚825 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7168615844864 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71067129 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44889147 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7146300523806 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103341677 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚375212193551360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚392818310093824 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚392818310093824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7168703866880 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103351820 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174121984 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚394872528813742 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚399987031000064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚387726211250576 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180869480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7168615844864 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚382591647407616 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚394872425625908 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚392818327133184 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7146214375332 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26178235 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚375212218455040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚399986925938688 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77527805 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚382591448381952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚400025075484160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚394910076833058 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚392855676461056 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77535124 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44893437 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚392855676461056 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71073887 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚392855693500416 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180896858 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚382627836217344 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103361732 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103351575 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7380136303616 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77535124 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚400025075484160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7379947508736 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7146976475362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180886699 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77535126 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚10199 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103361732 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚382628049915904 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7146890326776 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚375247913612288 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712ÅÛ'¥¨$™æM \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7146976475362 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 180886699 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77535126 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚10199 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103361732 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚382628049915904 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7146890326776 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚375247913612288 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚254 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103351575 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚392855693500416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26180707 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚400024970415616 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7169381983744 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7169293954560 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚394909973645112 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚400025075484160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚394910076833058 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚392855676461056 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77535124 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44893437 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚392855676461056 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71073887 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚392855693500416 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180896858 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚382627836217344 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103361732 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103351575 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7380136303616 RÈÁ"tDDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77535124 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚400025075484160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7379947508736ºÐÎ2š- _wandbruntime‚4162  _runtime‚4162ÍGÛ¾ ›ƒ 1loss/FBt2/assignment_loss‚0.07103477669307565 0loss/SRqqt1/detection_loss‚0.986566654689812 %loss/total_loss‚8.922605011900979 trainer/global_step‚10199 $ _timestamp‚1.784506008112445e+09 _runtime‚4162.175283998 2loss/SRqqt2/assignment_loss‚0.6195089053586221 /loss/FRt2/detection_loss‚1.4692620018828246 0loss/FRt1/assignment_loss‚2.0606291123633254 /loss/FRt1/detection_loss‚0.5214603360976633 /loss/FBt1/detection_loss‚0.7602377871051431 0loss/FBt2/detection_loss‚0.32964229766238373 epoch‚12 /loss/FRt2/assignment_loss‚0.734253064526452 2loss/SRqqt1/assignment_loss‚0.7183282720794655 3loss/FBt1/assignment_loss‚0.0053756165258899815 1loss/SRqqt2/detection_loss‚0.6463061869163217 _step‚827ï½Öœˆ 0loss/FRt1/assignment_loss‚2.0606291123633254 /loss/FRt1/detection_loss‚0.5214603360976633 /loss/FBt1/detection_loss‚0.7602377871051431 0loss/FBt2/detection_loss‚0.32964229766238373 epoch‚12 /loss/FRt2/assignment_loss‚0.734253064526452 2loss/SRqqt1/assignment_loss‚0.7183282720794655 3loss/FBt1/assignment_loss‚0.0053756165258899815 1loss/SRqqt2/detection_loss‚0.6463061869163217 _step‚827 1loss/FBt2/assignment_loss‚0.07103477669307565 0loss/SRqqt1/detection_loss‚0.986566654689812 %loss/total_loss‚8.922605011900979 trainer/global_step‚10199 $ _timestamp‚1.784506008112445e+09 _runtime‚4162.175283998 2loss/SRqqt2/assignment_loss‚0.6195089053586221 /loss/FRt2/detection_loss‚1.4692620018828246»¡BÃik‚(:à ŸÍõÒèžÕ¿ network.sent‚ 1.3515403e+07& disk./.usagePercent‚76.41474840754 disk.nvme1n1.in‚ 12.11328125 disk.md0.out‚138.5! disk.dm-1.out‚4086.0458984375 disk.rbd0.in‚0$ memory_percent‚99.21159744262695* proc.memory.percent‚17.526817321777344% disk./.usageGB‚1173.1574249267578" disk.nvme0c0n1.in‚ 129.26171875 disk.nvme0n1.in‚ 129.26171875& disk.nvme1c1n1.out‚12587.158203125 disk.md0.in‚0 disk.dm-0.in‚141.375# disk.dm-0.out‚43547.27783203125 disk.dm-1.in‚0 network.recv‚ 1.187139e+06& disk.nvme0n1.out‚35340.20263671875! disk.nvme1c1n1.in‚ 12.11328125 disk.rbd0.out‚0& proc.memory.availableMB‚ 129.171875 proc.memory.rssMB‚ 2871.59375( disk.nvme0c0n1.out‚35340.20263671875$ disk.nvme1n1.out‚12587.158203125 cpu‚26.43545551701171 proc.cpu.threads‚17(P’‘ž‚(:† ŸÍõÒ¹À gpu.0.gpu‚73.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚113.058' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.686 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±‰ª2Ÿ- _wandbruntime‚4181  _runtime‚4181m+wöÈ Â _runtime‚4181.298588958 _step‚828 & lr-AdamW/pg1‚9.684519239030376e-05 & lr-AdamW/pg2‚9.684519239030376e-05 trainer/global_step‚10249 % _timestamp‚1.7845060272337441e+09Œ§ž Í¡Ç & lr-AdamW/pg2‚9.684519239030376e-05 trainer/global_step‚10249 % _timestamp‚1.7845060272337441e+09 _runtime‚4181.298588958 _step‚828 & lr-AdamW/pg1‚9.684519239030376e-05¼F't¦2¢- _wandbruntime‚4181  _runtime‚4181ÿ ÌúÝ'£×O OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26302033 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚384411380280320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚389569831677936 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _runtime‚4181.298663398 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71405414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚401889182042112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77893853 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77893853 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚394686645494784 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77893855 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 % _timestamp‚1.7845060272343338e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚394686628455424 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77893855 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚376996976317440 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7414403962880 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7180129999214 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚376996951413760 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103848572 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103848572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45103644 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚389569814638576 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚401889287103488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7202553586688 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚384411162380288 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 181742427 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192996352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7414210966528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 103848572 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 103848572 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45103644 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚389569814638576 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚401889287103488 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7202553586688 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚384411162380288 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 181742427 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192996352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7414210966528 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71405414 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚401889182042112 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚77893853 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77893853 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚394686645494784½½æ¼2¥- _wandbruntime‚4181  _runtime‚4181âa_Y+ ¦àM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚401927331587584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45107937 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚396787509696466 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7180719802184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103858484 trainer/global_step‚10249 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103848327 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71412491 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77901176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚377032646571008 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45107934 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚377032671474688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚389606703745696 _step‚830 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚394723994822656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26304288 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103858484 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚401927331587584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7203319725568 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181759660 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 % _timestamp‚1.7845060274922779e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7203231696384 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181749501 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 4181.55560547 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181749501 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77901176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181759660 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71412222 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7415097129472 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚394724011862016 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7203231696384 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚384447554905600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77901174 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚384447768604160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77901174 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7180805950770 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚394724011862016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7414908334592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103848327 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157cÉWÜë&§åM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚394724011862016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7414908334592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 103848327 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚401927331587584 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45107937 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚396787509696466 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7180719802184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 103858484 trainer/global_step‚10249 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103848327 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71412491 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77901176 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚377032646571008 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45107934 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚377032671474688 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚389606703745696 _step‚830 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚394723994822656 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26304288 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 103858484 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚401927331587584 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7203319725568 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181759660 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 % _timestamp‚1.7845060274922779e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7203231696384 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181749501 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚ 4181.55560547 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181749501 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77901176 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181759660 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71412222 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7415097129472 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚394724011862016 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7203231696384 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚384447554905600 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77901174 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚384447768604160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77901174 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7180805950770 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712¾?RÂ2¨- _wandbruntime‚4181  _runtime‚4181~Õæó ©„ 2loss/SRqqt1/assignment_loss‚0.5772747180403164 2loss/FBt1/assignment_loss‚0.013652441970627913 /loss/FRt1/detection_loss‚0.5458185550245763 /loss/FBt1/detection_loss‚0.8190270111102749 trainer/global_step‚10249 /loss/FRt1/assignment_loss‚2.227083071912051 0loss/FRt2/assignment_loss‚1.0572864146825904 1loss/SRqqt1/detection_loss‚1.3406584507237511 0loss/FBt2/detection_loss‚0.44478446952876094 %loss/total_loss‚9.932493192563596 epoch‚12 2loss/SRqqt2/assignment_loss‚0.6151447846227994 1loss/FBt2/assignment_loss‚0.05080613744184379 1loss/SRqqt2/detection_loss‚0.6019017072220196 _runtime‚4181.556088721 _step‚831 /loss/FRt2/detection_loss‚1.6390554302839846 % _timestamp‚1.7845060274932396e+09þ"ôª‰ /loss/FRt1/detection_loss‚0.5458185550245763 /loss/FBt1/detection_loss‚0.8190270111102749 trainer/global_step‚10249 /loss/FRt1/assignment_loss‚2.227083071912051 0loss/FRt2/assignment_loss‚1.0572864146825904 1loss/SRqqt1/detection_loss‚1.3406584507237511 0loss/FBt2/detection_loss‚0.44478446952876094 %loss/total_loss‚9.932493192563596 epoch‚12 2loss/SRqqt2/assignment_loss‚0.6151447846227994 1loss/FBt2/assignment_loss‚0.05080613744184379 1loss/SRqqt2/detection_loss‚0.6019017072220196 _runtime‚4181.556088721 _step‚831 /loss/FRt2/detection_loss‚1.6390554302839846 % _timestamp‚1.7845060274932396e+09 2loss/SRqqt1/assignment_loss‚0.5772747180403164 2loss/FBt1/assignment_loss‚0.013652441970627913¿R\bÒ{«‚(:ð ®ÍõÒå”ò¿ proc.cpu.threads‚17 network.sent‚ 1.3577772e+07 disk.md0.out‚ 139.2265625 disk.dm-0.in‚141.375# disk.dm-0.out‚43550.87158203125( disk.nvme0c0n1.out‚35344.09716796875* proc.memory.percent‚17.526817321777344) disk./.usagePercent‚76.41475809802645% disk./.usageGB‚1173.1575736999512 disk.rbd0.in‚0 disk.rbd0.out‚0! disk.nvme1c1n1.in‚ 12.11328125 disk.nvme1n1.in‚ 12.11328125& disk.nvme1n1.out‚12588.44775390625# memory_percent‚99.2119550704956 disk.md0.in‚0" disk.dm-1.out‚4086.11279296875 disk.nvme0n1.in‚ 129.26171875 proc.memory.rssMB‚ 2871.59375 cpu‚26.434254200691008 network.recv‚ 1.191394e+06 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 129.26171875& disk.nvme0n1.out‚35344.09716796875( disk.nvme1c1n1.out‚12588.44775390625( proc.memory.availableMB‚ 129.11328125 ‚³$š¬‚(: ®ÍõÒ ÉÎÀ gpu.0.gpu‚65.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚195.11' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.03666666666668 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RÙ]v­‚(:ë ½ÍõÒŸÔØÀ& disk.nvme0n1.out‚35360.37255859375 disk.rbd0.in‚0 disk.dm-0.in‚141.375$ memory_percent‚99.21209812164307 proc.cpu.threads‚17 disk.md0.out‚ 143.12890625 network.recv‚ 1.195838e+06$ disk./.usageGB‚1173.157470703125 disk.dm-0.out‚ 43563.3046875" disk.dm-1.out‚4086.21435546875( proc.memory.availableMB‚ 129.08984375 proc.memory.rssMB‚ 2871.59375* proc.memory.percent‚17.526817321777344! disk.nvme1c1n1.in‚ 12.11328125 disk.nvme1n1.in‚ 12.11328125% disk.nvme1n1.out‚12592.5869140625 disk.md0.in‚0 disk.dm-1.in‚0 disk.rbd0.out‚0 cpu‚26.432519936753785' disk.nvme1c1n1.out‚12592.5869140625 network.sent‚ 1.3642175e+07) disk./.usagePercent‚76.41475138922814" disk.nvme0c0n1.in‚ 129.26171875( disk.nvme0c0n1.out‚35360.37255859375 disk.nvme0n1.in‚ 129.26171875ïHš®‚(: ½ÍõÒ„œ‰Á gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.67' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.223333333333336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sHM2¯- _wandbruntime‚4200  _runtime‚4200žF‹cÈ°Â & lr-AdamW/pg1‚9.669090265290278e-05 & lr-AdamW/pg2‚9.669090265290278e-05 trainer/global_step‚10299 % _timestamp‚1.7845060462405367e+09 _runtime‚4200.303671645 _step‚832sJp¯Í±Ç & lr-AdamW/pg1‚9.669090265290278e-05 & lr-AdamW/pg2‚9.669090265290278e-05 trainer/global_step‚10299 % _timestamp‚1.7845060462405367e+09 _runtime‚4200.303671645 _step‚832ÀŒ`¾ú2²- _wandbruntime‚4200  _runtime‚4200:8èÝ'³×O VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78259905 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚386230900667904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7213959474622 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚398627291352724 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚403791543206912 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78259903 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚391413418026576 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚396554963856384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182605229 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7236579350528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104335181 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45318147 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104345324 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403791438145536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚10299 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78259903 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚396554946817024 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104345324 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78259905 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚403791438145536 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7213873326148 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚391413435065936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚403791543206912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7236579350528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7236491328512 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182595084 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26426028 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26425777 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 % _timestamp‚1.7845060462414024e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚403791438145536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 trainer/global_step‚10299 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78259903 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚396554946817024 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104345324 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78259905 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚403791438145536 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7213873326148 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚391413435065936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚403791543206912 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7236579350528 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7236491328512 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 182595084 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26426028 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26425777 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 % _timestamp‚1.7845060462414024e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚403791543206912 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78259903 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚391413418026576 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚396554963856384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 182605229 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7236579350528 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145Áw¡Ôà2µ- _wandbruntime‚4200  _runtime‚4200C­ç&¶áM FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7237257467392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71750654 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 trainer/global_step‚10299 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚398664839371928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚403829482622464 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182622462 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7237257467392 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104345079 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚391450307133696 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104345079 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78267226 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7450075256832 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7449888559104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403829587691008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7214635426178 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚396592313184256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182622462 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182612303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚386267292992512 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 % _timestamp‚1.7845060464983382e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45322434 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7214549277592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚396592330223616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45322437 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚396592330223616 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7237169438208 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71750914 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152èò72ì&·æM XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7214549277592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚396592330223616 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45322437 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚396592330223616 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7237169438208 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚71750914 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7237257467392 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71750654 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 trainer/global_step‚10299 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚398664839371928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚403829482622464 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 182622462 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7237257467392 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104345079 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚391450307133696 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104345079 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78267226 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7450075256832 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7449888559104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚403829587691008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7214635426178 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚396592313184256 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 182622462 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 182612303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚260 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚386267292992512 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 % _timestamp‚1.7845060464983382e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45322434ÂõU˜ž2¸- _wandbruntime‚4200  _runtime‚4200ñ%ûê ¹ƒ 3loss/FBt1/assignment_loss‚0.0016899363456412964 1loss/SRqqt1/detection_loss‚1.1721214469340022 trainer/global_step‚10299 _step‚835 /loss/FRt1/assignment_loss‚2.079104979460438 .loss/FRt1/detection_loss‚0.575586961342583 .loss/FRt2/detection_loss‚1.546056804715601 0loss/FRt2/assignment_loss‚0.8940916786006853 2loss/SRqqt1/assignment_loss‚0.6742333532201356 4loss/FBt2/assignment_loss‚2.2983042436861957e-05 /loss/SRqqt2/detection_loss‚0.56531349629062 /loss/FBt2/detection_loss‚0.6100263600314005 _runtime‚4200.562431649 2loss/SRqqt2/assignment_loss‚0.5943891348766073 /loss/FBt1/detection_loss‚0.7492359288735315 %loss/total_loss‚9.461873063733684 epoch‚12 % _timestamp‚1.7845060464992864e+09„¯Ö§ºˆ 2loss/SRqqt2/assignment_loss‚0.5943891348766073 /loss/FBt1/detection_loss‚0.7492359288735315 %loss/total_loss‚9.461873063733684 epoch‚12 % _timestamp‚1.7845060464992864e+09 3loss/FBt1/assignment_loss‚0.0016899363456412964 1loss/SRqqt1/detection_loss‚1.1721214469340022 trainer/global_step‚10299 _step‚835 /loss/FRt1/assignment_loss‚2.079104979460438 .loss/FRt1/detection_loss‚0.575586961342583 .loss/FRt2/detection_loss‚1.546056804715601 0loss/FRt2/assignment_loss‚0.8940916786006853 2loss/SRqqt1/assignment_loss‚0.6742333532201356 4loss/FBt2/assignment_loss‚2.2983042436861957e-05 /loss/SRqqt2/detection_loss‚0.56531349629062 /loss/FBt2/detection_loss‚0.6100263600314005 _runtime‚4200.562431649ÃöªŠ¿‘»‚(:† ÌÍõÒœî…À gpu.0.gpu‚82.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚201.075' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚67.025 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ðî•/z¼‚(:ï ÌÍõҨɚÀ% disk./.usageGB‚1173.1574935913086( disk.nvme1c1n1.out‚12593.92431640625 disk.md0.in‚0$ memory_percent‚99.21393394470215 proc.memory.rssMB‚ 2871.59375) disk./.usagePercent‚76.41475288007221& disk.nvme0n1.out‚35382.66455078125! disk.nvme1c1n1.in‚ 12.11328125& disk.nvme1n1.out‚12593.92431640625 disk.dm-1.in‚0* proc.memory.percent‚17.526817321777344 cpu‚26.430814881119716 network.recv‚ 1.199471e+06 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 129.26171875( disk.nvme0c0n1.out‚35382.66455078125 disk.nvme0n1.in‚ 129.26171875" disk.dm-1.out‚4086.31396484375 network.sent‚ 1.3658041e+07 disk.nvme1n1.in‚ 12.11328125 disk.md0.out‚ 143.515625 disk.dm-0.in‚141.375# disk.dm-0.out‚43585.99951171875' proc.memory.availableMB‚ 128.7890625 proc.cpu.threads‚17æ+’F2½- _wandbruntime‚4219  _runtime‚4219_…úuȾ trainer/global_step‚10349 % _timestamp‚1.7845060652395172e+09 _runtime‚4219.302702487 _step‚836 & lr-AdamW/pg1‚9.653596218531555e-05 & lr-AdamW/pg2‚9.653596218531555e-05D¸:2Í¿Ç trainer/global_step‚10349 % _timestamp‚1.7845060652395172e+09 _runtime‚4219.302702487 _step‚836 & lr-AdamW/pg1‚9.653596218531555e-05 & lr-AdamW/pg2‚9.653596218531555e-05Ä•ÆE2À- _wandbruntime‚4219  _runtime‚4219;&¥¾Á×O G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45532647 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183457886 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚393257038453936 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚393257021414576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104831933 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚405693694248960 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78625955 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚405693799310336 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚405693694248960 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104831933 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚4219.303695878 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78625953 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7270517092352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183468031 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78625953 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚398423282217984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚837 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72081938 trainer/global_step‚10349 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚400504724216132 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7270517092352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26549294 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_startUL#þÈ/requested_bytes.large_pool.peak‚ 5805303808 % _timestamp‚1.7845060652405593e+09 JI«â'ÂÜO BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45532647 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183457886 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚393257038453936 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚393257021414576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104831933 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚405693694248960 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78625955 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚405693799310336 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚405693694248960 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104831933 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚4219.303695878 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78625953 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7270517092352 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183468031 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78625953 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚398423282217984 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚837 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72081938 trainer/global_step‚10349 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚400504724216132 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7270517092352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚251 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26549294 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 % _timestamp‚1.7845060652405593e+09 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚398460631545856 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚388087034322944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78633276 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104841831 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7271195209216 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72088989 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚405731738725888 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78633274 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104851988 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183485264 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚10349 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7271107180032 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚400542375423282 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7248378753000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚398460648585216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚405731843794432 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚398460631545856 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72088736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45536937 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚400542272235336 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45536934 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0¹ï¦}ÅæM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26552052 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _step‚838 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚388087241730048 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7271195209216 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26551802 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7485054530560 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7271107180032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚398460648585216 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45536934 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚380602187199488 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚398460631545856 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚388087034322944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78633276 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 104841831 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7271195209216 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72088989 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚405731738725888 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78633274 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104851988 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183485264 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚10349 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7271107180032 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚400542375423282 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7248378753000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚398460648585216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚405731843794432 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚250 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_trå¸áOÒ ain_batch_end/segment.large_pool.peak‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚398460631545856 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚253 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72088736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45536937 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚400542272235336 PFÈŠ _runtime‚4219.564525794 2loss/SRqqt1/assignment_loss‚0.6009309459333962 3loss/FBt1/assignment_loss‚0.0008899707585277551 1loss/SRqqt1/detection_loss‚0.8920837298617699 1loss/SRqqt2/detection_loss‚0.6299655582404767 0loss/FBt2/detection_loss‚0.40188965314953795 trainer/global_step‚10349 _step‚839 3loss/FBt2/assignment_loss‚0.0045433970962779485 /loss/FRt1/detection_loss‚0.5325826632725144 -loss/FRt2/detection_loss‚1.76500186482661 %loss/total_loss‚9.295859212875968 epoch‚12 /loss/FRt1/assignment_loss‚2.107472272084491 2loss/SRqqt2/assignment_loss‚0.6289177028003794 /loss/FBt1/detection_loss‚0.7132501447682872 % _timestamp‚1.7845060655016234e+09 0loss/FRt2/assignment_loss‚1.0183313100836997Ç=Þ×uÉ‚(:ê ÛÍõÒ»êé¿ network.sent‚ 1.3707806e+07 disk.dm-1.in‚0" disk.dm-1.out‚4086.41162109375 disk.rbd0.in‚0 disk.nvme0n1.in‚ 129.26171875 disk.md0.in‚0$ memory_percent‚99.21433925628662( proc.memory.availableMB‚ 128.72265625' disk.nvme1c1n1.out‚12594.4189453125 network.recv‚ 1.203481e+06' disk./.usagePercent‚76.414727287249% disk./.usageGB‚1173.1571006774902 disk.rbd0.out‚0& disk.nvme0c0n1.out‚35383.986328125! disk.nvme1c1n1.in‚ 12.11328125% disk.nvme1n1.out‚12594.4189453125 disk.nvme1n1.in‚ 12.11328125 disk.md0.out‚ 143.80078125 disk.dm-0.in‚141.375# disk.dm-0.out‚43587.08837890625* proc.memory.percent‚17.526817321777344 cpu‚26.429297267185724" disk.nvme0c0n1.in‚ 129.26171875$ disk.nvme0n1.out‚35383.986328125 proc.memory.rssMB‚ 2871.59375 proc.cpu.threads‚17 12šÊ‚(: ÛÍõÒü”ÇÀ gpu.0.gpu‚66.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚200.0' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.66666666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0`Ùφ2Ë- _wandbruntime‚4237  _runtime‚4237Ø°,ÔÆÌÀ _runtime‚4237.941725309 _step‚840 % lr-AdamW/pg1‚9.63803734692282e-05 % lr-AdamW/pg2‚9.63803734692282e-05 trainer/global_step‚10399 % _timestamp‚1.7845060838785794e+09®„ËÍÅ % lr-AdamW/pg1‚9.63803734692282e-05 % lr-AdamW/pg2‚9.63803734692282e-05 trainer/global_step‚10399 % _timestamp‚1.7845060838785794e+09 _runtime‚4237.941725309 _step‚840Èü@dY2Î- _wandbruntime‚4237  _runtime‚4237ü\ÂOÝ'Ï×O GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105328685 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚389870353320960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45747144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184320688 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72419668 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚389870556540928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78992003 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7304454834176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 trainer/global_step‚10399 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚407595950352384 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚400291583540224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚395100641841936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105338828 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7519306636288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7281618425438 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7304454834176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26672767 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚400291583540224 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚407596055413760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚402382157079540 % _timestamp‚1.7845060838794456e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚382351249904640 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7519128320000 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚395100624802576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚400291600579584 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105338828 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚402382260267374 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7281532276964 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78992005 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚407596055413760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78992003 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 _step‚841 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7304366812160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78992005 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚382351225000960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚407595950352384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45747147 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152)\â'ÐÜO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105328685 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚389870353320960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45747144 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 203219968 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 184320688 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72419668 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚389870556540928 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78992003 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7304454834176 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 trainer/global_step‚10399 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚407595950352384 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚400291583540224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚395100641841936 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105338828 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7519306636288 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7281618425438 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7304454834176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26672767 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚400291583540224 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚407596055413760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚402382157079540 % _timestamp‚1.7845060838794456e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚382351249904640 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7519128320000 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚395100624802576 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚400291600579584 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105338828 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚402382260267374 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7281532276964 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78992005 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚407596055413760 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78992003 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 _step‚841 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7304366812160 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78992005 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚382351225000960 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚407595950352384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45747147 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016ÉÔ]v–2Ñ- _wandbruntime‚4238  _runtime‚4238®RŠÒàM FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72426319 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184337907 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78999324 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚407634099897856 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7282208228408 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚400328966946816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26674885 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚402419808286690 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105348740 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184348066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7305132951040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7305044921856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105338583 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚382386945061888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚395137513909696 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚400328949907456 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105348740 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _runtime‚4238.199506127 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7305044921856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚400328949907456 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 $ _timestamp‚1.784506084136135e+09 _step‚842 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚389906741123584 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7519820965376 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105338583 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚382386920158208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚400328966946816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚407633994829312 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78999324V­Të&ÓåM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45751437 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚395137496870336 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989184 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72426319 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 184337907 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78999324 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚407634099897856 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7282208228408 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚400328966946816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26674885 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚402419808286690 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105348740 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 184348066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7305132951040 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7305044921856 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105338583 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚382386945061888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚395137513909696 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚400328949907456 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105348740 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 _runtime‚4238.199506127 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7305044921856 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚400328949907456 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚251 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 $ _timestamp‚1.784506084136135e+09 _step‚842 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚389906741123584 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7519820965376 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105338583 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚382386920158208 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚400328966946816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚407633994829312 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78999324 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 F&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1071428582072258 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5991309285163879 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7549173831939697 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998833417892456 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9946419596672058 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9856441020965576 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978741407394409 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 )jet/accuracy_1_of_2‚0.793864905834198 +particle/specificity‚0.9149893522262573 epoch‚12 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7745732665061951 *jet/accuracy_2_of_2‚0.1582464873790741 particle/accuracy_2_of_4‚1 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13225741684436798 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962481260299683 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9967128038406372 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6069287061691284 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.696060061454773 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6038928031921387 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07572811841964722 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29612040519714355 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6923333406448364 7validation_average_jet_accuracy‚0.38053858280181885 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5535070896148682 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05008630082011223 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1071428582072258 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23101919889450073 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03210978955030441 particle/accuracy_1_of_3‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8615384101867676 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.25193244218826294 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6758701801300049 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19193914532661438 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8572224378585815 (particle/accuracy‚0.8536666631698608 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8609023094177246 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.773217499256134 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9901729822158813 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21134094893932343 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7773504257202148 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1164366751909256 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7431843876838684 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9951766133308411 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8582722544670105 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9921368360519409 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10000000149011612 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9864532947540283 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4318181872367859 *jet/accuracy_1_of_3‚0.9278928637504578 ,jet/accuracy_4_of_4‚0.014682539738714695 jet/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2241673469543457 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.993835985660553 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6620741486549377 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8572224378585815 trainer/global_step‚10412 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8100000023841858 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.949999988079071 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958561658859253 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9980574250221252 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7703768610954285 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19208595156669617 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8100000023841858 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9464285969734192 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21008098125457764 particle/accuracy_3_of_3‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21991264820098877 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22894100844860077 +particle/sensitivity‚0.6294333338737488 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10000000149011612 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26449012756347656 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7346252799034119 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.49856439232826233 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20363391935825348 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7182105779647827 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9781084656715393 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6634573936462402 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0881490632891655 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23393644392490387 jet/accuracy_5_of_5‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03210978955030441 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24306133389472961 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4626719653606415 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.42308029532432556 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 particle/accuracy_3_of_5‚1 % _timestamp‚1.7845061045912523e+09 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +jet/accuracy_1_of_1‚0.33914533257484436 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.653217613697052 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.855570912361145 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3312750458717346 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5324409604072571 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8753958940505981 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7584733963012695 particle/accuracy_1_of_1‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5319987535476685 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02905805967748165 )jet/accuracy_2_of_4‚0.820858895778656 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39751380681991577 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.41979900002479553 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8123326301574707 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4517720639705658 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.811174750328064 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7496462464332581 particle/accuracy_1_of_2‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.18651016056537628 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.711999237537384 7"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.70933997631073 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03178348392248154 )loss/val_total_loss‚8.789816124096697 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24232308566570282 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.939303457736969 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10058895498514175 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7034403681755066 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6708886027336121 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7629255652427673 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3194444477558136 _step‚844 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23084786534309387 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7767117619514465 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5798611044883728 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24222171306610107 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0460246242582798 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_2_of_3‚0.5779950022697449 /particle/accuracy_3_of_4‚0.9947671294212341 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.29329782724380493 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8953068256378174 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04042123630642891 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 'particle/f_score‚0.6487630009651184 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.588416576385498 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3858024477958679 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6955825686454773 particle/accuracy_2_of_3‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚0 _runtime‚4258.654845416 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.48465025424957275 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 jet/accuracy_4_of_5‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5404460430145264 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2240811288356781 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02933340333402157 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8469298481941223 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03909851610660553 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5169271230697632 particle/accuracy_4_of_5‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6749313473701477 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 ö¹92ÛÂ{ :"Purity/*FRt1SRqqt*FBt/event_purity‚0.39751380681991577 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.41979900002479553 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8123326301574707 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.04624721035ƒéÁ–<36129 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4517720639705658 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.811174750328064 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7496462464332581 particle/accuracy_1_of_2‚1 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.18651016056537628 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.711999237537384 7"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.70933997631073 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.03178348392248154 )loss/val_total_loss‚8.789816124096697 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.24232308566570282 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.939303457736969 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.10058895498514175 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.7034403681755066 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6708886027336121 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7629255652427673 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.3194444477558136 _step‚844 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.23084786534309387 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7767117619514465 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.5798611044883728 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.24222171306610107 9"Purity/2FRt*SRqqt*FBt/event_purity‚0.0460246242582798 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 *jet/accuracy_2_of_3‚0.5779950022697449 /particle/accuracy_3_of_4‚0.9947671294212341 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.29329782724380493 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8953068256378174 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.04042123630642891 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 'particle/f_score‚0.6487630009651184 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 8"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.588416576385498 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3858024477958679 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6955825686454773 particle/accuracy_2_of_3‚1 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_5‚1 particle/accuracy_5_of_5‚0 _runtime‚4258.654845416 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.48465025424957275 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 jet/accuracy_4_of_5‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5404460430145264 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.2240811288356781 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.02933340333402157 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8469298481941223 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 :"Purity/2FRt*SRqqt0FBt/event_purity‚0.03909851610660553 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.5169271230697632 particle/accuracy_4_of_5‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6749313473701477 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 /particle/accuracy_4_of_4‚0.9442457556724548 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 particle/accuracy_2_of_5‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9960257411003113 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8521930575370789 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.48500001430511475 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9189749956130981 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6708886027336121 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.17644813656806946 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.39723050594329834 +jet/accuracy_3_of_4‚0.10087835043668747 particle/accuracy_1_of_4‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6885349750518799 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5311824083328247 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 jet/accuracy_2_of_5‚1 +jet/accuracy_3_of_3‚0.06969700753688812 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6017816662788391 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.11565429717302322 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7589041590690613 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.13124999403953552 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.23084786534309387 *jet/accuracy_1_of_4‚0.9661092162132263 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.0321071483194828 6 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.989054262638092 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.19385507702827454 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20431189239025116 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6683235764503479 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1071428582072258 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5991309285163879 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7549173831939697 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 6 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.998833417892456 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9946419596672058 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9856441020965576 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9978741407394409 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 )jet/accuracy_1_of_2‚0.793864905834198 +particle/specificity‚0.9149893522262573 epoch‚12 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7745732665061951 *jet/accuracy_2_of_2‚0.1582464873790741 particle/accuracy_2_of_4‚1 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.13225741684436798 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9962481260299683 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9967128038406372 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6069287061691284 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.696060061454773 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6038928031921387 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 :"Purity/2FRt0SRqqt0FBt/event_purity‚0.07572811841964722 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.29612040519714355 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6923333406448364 7validation_average_jet_accuracy‚0.38053858280181885 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5535070896148682 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.05008630082011223 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1071428582072258 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.23101919889450073 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.03210978955030441 particle/accuracy_1_of_3‚1 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8615384101867676 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.25193244218826294 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6758701801300049 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.19193914532661438 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8572224378585815 (particle/accuracy‚0.8536666631698608 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8609023094177246 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 8"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.773217499256134 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 jet/accuracy_3_of_5‚1 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9901729822158813 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.21134094893932343 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7773504257202148 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1164366751909256 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7431843876838684 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9951766133308411 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8582722544670105 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9921368360519409 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10000000149011612 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9864532947540283 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4318181872367859 *jet/accuracy_1_of_3‚0.9278928637504578 ,jet/accuracy_4_of_4‚0.014682539738714695 jet/accuracy_1_of_5‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.2241673469543457 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.993835985660553 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6620741486549377 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8572224378585815 trainer/global_step‚10412 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8100000023841858 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.949999988079071 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9958561658859253 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9980574250221252 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7703768610954285 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_2‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.19208595156669617 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8100000023841858 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9464285969734192 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.21008098125457764 particle/accuracy_3_of_3‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.21991264820098877 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.22894100844860077 +particle/sensitivity‚0.6294333338737488 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10000000149011612 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.26449012756347656 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7346252799034119 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.49856439232826233 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20363391935825348 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7182105779647827 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9781084656715393 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6634573936462402 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0881490632891655 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.23393644392490387 jet/accuracy_5_of_5‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.03210978955030441 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24306133389472961 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4626719653606415 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.42308029532432556 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 particle/accuracy_3_of_5‚1 % _timestamp‚1.7845061045912523e+09 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 +jet/accuracy_1_of_1‚0.33914533257484436 8"Purity/0FRt*SRqqt0FBt/event_purity‚0.653217613697052 8"Purity/0FRt*SRqqt1FBt/event_purity‚0.855570912361145 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.3312750458717346 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5324409604072571 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8753958940505981 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7584733963012695 particle/accuracy_1_of_1‚1 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5319987535476685 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.02905805967748165 )jet/accuracy_2_of_4‚0.820858895778656Ì-C5qÜ‚(:æ ùÍõÒ´çò¿" proc.memory.rssMB‚ 2874.1796875 network.recv‚ 1.212118e+06& disk.nvme1c1n1.out‚12659.533203125$ disk.nvme1n1.out‚12659.533203125 disk.dm-1.in‚0 disk.dm-1.out‚ 4150.421875$ memory_percent‚98.82626533508301 proc.cpu.threads‚17( disk.nvme0c0n1.out‚35400.98779296875& disk.nvme0n1.out‚35400.98779296875 disk.dm-0.in‚141.375 cpu‚26.43757204947592$ disk./.usageGB‚1173.407642364502" disk.nvme0c0n1.in‚ 129.26171875 disk.nvme0n1.in‚ 129.26171875 disk.nvme1n1.in‚ 12.11328125# disk.dm-0.out‚43604.20458984375 disk.rbd0.in‚0* proc.memory.percent‚17.542600631713867 network.sent‚ 1.3833593e+07( disk./.usagePercent‚76.4310465633937! disk.nvme1c1n1.in‚ 12.11328125 disk.md0.in‚0 disk.md0.out‚ 144.2421875 disk.rbd0.out‚0' proc.memory.availableMB‚ 192.3046875“Jƒ^Ý‚(:„ ùÍõÒÆŒÀ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.171' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚37.057 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~rÎþ2Þ- _wandbruntime‚4273  _runtime‚4273 ‰Èß trainer/global_step‚10449 % _timestamp‚1.7845061194457626e+09 _runtime‚4273.508912282 _step‚845 & lr-AdamW/pg1‚9.622413899670978e-05 & lr-AdamW/pg2‚9.622413899670978e-05^òÇŠÍàÇ _runtime‚4273.508912282 _step‚845 & lr-AdamW/pg1‚9.622413899670978e-05 & lr-AdamW/pg2‚9.622413899670978e-05 trainer/global_step‚10449 % _timestamp‚1.7845061194457626e+09Í]¼ò72á- _wandbruntime‚4273  _runtime‚4273)&®FÝ'â×O H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105992703 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46055724 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚410491290368000 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚403132565496832 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185598369 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79595521 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚385061406127104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 185588224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106002846 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚405248576606700 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚392636219542016 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚405248473418866 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚410491395429376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚392636005836288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚385061431030784 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26877653 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _step‚846 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚403132582536192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚403132565496832 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46055727 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7574788511232 % _timestamp‚1.7845061194466314e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106002846 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7358812893184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72933377 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257Ǭþ#UãÜO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚410491395429376 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚72933637 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚10449 _runtime‚4273.509773264 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7358812893184 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7358724871168 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚397912734698288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚403132582536192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚403132565496832 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMAâ,Conitor.on_train_batch_start/inactive_split.large_pool.allocated‚46055727 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7574788511232 % _timestamp‚1.7845061194466314e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106002846 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7358812893184 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72933377 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105992703 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46055724 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚410491290368000 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚403132565496832 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185598369 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79595521 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚260 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚385061406127104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 185588224 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106002846 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚405248576606700 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚392636219542016 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚405248473418866 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚410491395429376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚392636005836288 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚385061431030784 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26877653 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 _step‚846 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185615602 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79602842 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚403169948903424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚397949606766048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7359491010048 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79602842 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 trainer/global_step‚10449 _step‚847 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72940407 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72940165 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚405286021438070 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79602844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚403169948903424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚403169931864064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26880390 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7575303958528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚385097101284352 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚410529439913472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79602844 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚392672405242880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185615602 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 % _timestamp‚1.7845061197059052e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚397949589726688 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7359491010048 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106012758 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 _runtime‚4273.769240383 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚1521¥krì&ææM REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚410529439913472 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚239 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79602844 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚392672405242880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 185615602 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 % _timestamp‚1.7845061197059052e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚397949589726688 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7359491010048 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106012758 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 _runtime‚4273.769240383 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7359402980864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185615602 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79602842 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚403169948903424 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚397949606766048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7359491010048 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚79602842 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 trainer/global_step‚10449 _step‚847 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72940407 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72940165 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚405286021438070 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79602844 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚403169948903424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚403169931864064 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26880390 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7575303958528 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚242 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚385097101284352 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751ϵRÀ¢2ç- _wandbruntime‚4273  _runtime‚4273 *‡ è„ 0loss/FRt2/assignment_loss‚0.9137402572061705 2loss/SRqqt2/assignment_loss‚0.4903252881737544 /loss/FRt1/detection_loss‚0.5594679532903934 0loss/SRqqt2/detection_loss‚0.605205894960222 epoch‚13 trainer/global_step‚10449 $ _timestamp‚1.784506119706856e+09 _runtime‚4273.769681684 2loss/SRqqt1/assignment_loss‚0.6050460658816647 4loss/FBt2/assignment_loss‚0.00026595083030633664 /loss/FRt2/detection_loss‚1.3920560648860545 .loss/FBt2/detection_loss‚0.589559957865992 3loss/FBt1/assignment_loss‚0.0005252432482999817 /loss/FBt1/detection_loss‚0.8278275778672347 %loss/total_loss‚9.081245751949007 /loss/FRt1/assignment_loss‚2.008453001477897 1loss/SRqqt1/detection_loss‚1.0887724962610175 _step‚848e‹é‰ 3loss/FBt1/assignment_loss‚0.0005252432482999817 /loss/FBt1/detection_loss‚0.8278275778672347 %loss/total_loss‚9.081245751949007 /loss/FRt1/assignment_loss‚2.008453001477897 1loss/SRqqt1/detection_loss‚1.0887724962610175 _step‚848 0loss/FRt2/assignment_loss‚0.9137402572061705 2loss/SRqqt2/assignment_loss‚0.4903252881737544 /loss/FRt1/detection_loss‚0.5594679532903934 0loss/SRqqt2/detection_loss‚0.605205894960222 epoch‚13 trainer/global_step‚10449 $ _timestamp‚1.784506119706856e+09 _runtime‚4273.769681684 2loss/SRqqt1/assignment_loss‚0.6050460658816647 4loss/FBt2/assignment_loss‚0.00026595083030633664 /loss/FRt2/detection_loss‚1.3920560648860545 .loss/FBt2/detection_loss‚0.589559957865992ÐI‘5›ê‚(: ˆÎõÒßÓ‚À gpu.0.gpu‚45.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.641' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.21366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Àpsë‚(:è ˆÎõÒ×÷–À disk.dm-0.out‚43607.37890625 disk.nvme0n1.in‚ 129.26171875 disk.dm-0.in‚141.375$ memory_percent‚98.82628917694092 cpu‚26.440467952060022 proc.cpu.threads‚17$ disk./.usageGB‚1173.407642364502% disk.nvme0n1.out‚35404.1982421875 disk.nvme1n1.in‚ 12.11328125 disk.md0.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 129.26171875' disk.nvme0c0n1.out‚35404.1982421875( proc.memory.availableMB‚ 192.30078125 network.recv‚ 1.216448e+06( disk./.usagePercent‚76.4310465633937! disk.nvme1c1n1.in‚ 12.11328125& disk.nvme1c1n1.out‚12660.419921875$ disk.nvme1n1.out‚12660.419921875 disk.md0.out‚ 144.64453125 disk.dm-1.in‚0" proc.memory.rssMB‚ 2874.1796875 network.sent‚ 1.3902322e+07! disk.dm-1.out‚4150.4912109375 disk.rbd0.out‚0* proc.memory.percent‚17.542600631713867ºyëþwì‚(:ì —ÎõÒâýþ¿$ memory_percent‚98.82798194885254' proc.memory.availableMB‚ 192.0234375 network.sent‚ 1.3953693e+07 network.recv‚ 1.220691e+06) disk./.usagePercent‚76.47449621993503" disk.nvme0c0n1.in‚ 129.30859375& disk.nvme0c0n1.out‚35629.236328125 disk.nvme0n1.in‚ 129.30859375! disk.nvme1c1n1.in‚ 12.11328125 disk.md0.out‚ 145.14453125% disk./.usageGB‚1174.0747032165527 disk.dm-0.out‚43877.88671875' disk.nvme1c1n1.out‚12770.9580078125 disk.nvme1n1.in‚ 12.11328125 disk.md0.in‚0* proc.memory.percent‚17.542600631713867 proc.cpu.threads‚17 disk.dm-0.in‚ 141.421875 disk.dm-1.in‚0" proc.memory.rssMB‚ 2874.1796875 cpu‚26.443465901932374! disk.dm-1.out‚4214.4990234375 disk.rbd0.in‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚35629.236328125% disk.nvme1n1.out‚12770.9580078125µ¦[‹›í‚(: —ÎõÒóù¯À gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.598' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.199333333333335 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V:SÓ2î- _wandbruntime‚4292  _runtime‚4292[?)hÆïÀ % lr-AdamW/pg1‚9.60672612701725e-05 % lr-AdamW/pg2‚9.60672612701725e-05 trainer/global_step‚10499 % _timestamp‚1.7845061387696135e+09 _runtime‚4292.832771286 _step‚849õ|©ËðÅ % lr-AdamW/pg1‚9.60672612701725e-05 % lr-AdamW/pg2‚9.60672612701725e-05 trainer/global_step‚10499 % _timestamp‚1.7845061387696135e+09 _runtime‚4292.832771286 _step‚849ÑÎJað2ñ- _wandbruntime‚4292  _runtime‚4292ã<Æž ò×O ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26999964 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚412393546471424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚850 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186461171 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79961571 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7609552946688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚412393546471424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7392662612992 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73270191 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79961573 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚394455716936192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ó À?79961571 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79961573 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106489455 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚405000883858432 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚407126009470108 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚399756338086288 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7369585235346 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 trainer/global_step‚10499 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚405000900897792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106499598 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79961573 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚257 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚394455716936192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚79961571 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79961573 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106489455 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚405000883858432 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚407126009470108 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚399756338086288 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7369585235346 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 trainer/global_step‚10499 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚405000900897792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106499598 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚412393546471424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚850 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 186461171 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79961571 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7609552946688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚412393546471424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7392662612992 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73270191 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238Ò‚ت=2ô- _wandbruntime‚4293  _runtime‚4293;Çéç&õáM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79968894 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27002089 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚405038267265024 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7370261186790 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106499353 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7393428751872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7393428751872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27002343 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 _runtime‚4293.423406585 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73276603 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79968894 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73276860 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46274517 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7370347335376 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106509510 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚386881884050432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 _step‚851 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚394492319891456 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚399793193114688 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚412431696016896 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106509510 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79968892 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚405038250225664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7393340722688 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚399793210154048 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186468245 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 % _timestamp‚1.7845061393600664e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186478404 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚394492110387200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7610251240448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7393340722688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚407163557489424 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚412431590948352 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46274514 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚405038250225664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186468245 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚386881859146752 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚10499%Œ£öæM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚79968894 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27002089 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚405038267265024 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7370261186790 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 106499353 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7393428751872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7393428751872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27002343 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 _runtime‚4293.423406585 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73276603 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79968894 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73276860 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46274517 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7370347335376 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106509510 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚386881884050432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 _step‚851 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚394492319891456 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚399793193114688 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚412431696016896 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106509510 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79968892 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚405038250225664 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7393340722688 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚399793210154048 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E?DeviceStatsMonitor.on_train_batch_end/active.large_³¨ˆÌIpool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186468245 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 % _timestamp‚1.7845061393600664e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186478404 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚394492110387200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7610251240448 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7393340722688 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚407163557489424 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚412431590948352 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46274514 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚405038250225664 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186468245 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚386881859146752 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚10499Óö¯3w2÷- _wandbruntime‚4293  _runtime‚4293òŸ"¬ ø„ trainer/global_step‚10499 $ _timestamp‚1.784506139361107e+09 3loss/FBt2/assignment_loss‚0.0001796581248727749 /loss/FRt1/detection_loss‚0.5241084449637391 1loss/SRqqt1/detection_loss‚0.9007748327352793 /loss/FBt2/detection_loss‚1.0900600420319717 _step‚852 0loss/FRt2/assignment_loss‚1.0165042357444762 2loss/SRqqt1/assignment_loss‚0.7079769445707299 /loss/FRt1/assignment_loss‚1.968496142711836 2loss/SRqqt2/assignment_loss‚0.6393284903779406 2loss/FBt1/assignment_loss‚0.004948007426791652 /loss/FRt2/detection_loss‚1.4372459924370051 1loss/SRqqt2/detection_loss‚0.6373274362570049 epoch‚13 _runtime‚4293.424280677 /loss/FBt1/detection_loss‚0.7684178642359458 %loss/total_loss‚9.695368091617594ù¥ÕPù‰ 1loss/SRqqt2/detection_loss‚0.6373274362570049 epoch‚13 _runtime‚4293.424280677 /loss/FBt1/detection_loss‚0.7684178642359458 %loss/total_loss‚9.695368091617594 trainer/global_step‚10499 $ _timestamp‚1.784506139361107e+09 3loss/FBt2/assignment_loss‚0.0001796581248727749 /loss/FRt1/detection_loss‚0.5241084449637391 1loss/SRqqt1/detection_loss‚0.9007748327352793 /loss/FBt2/detection_loss‚1.0900600420319717 _step‚852 0loss/FRt2/assignment_loss‚1.0165042357444762 2loss/SRqqt1/assignment_loss‚0.7079769445707299 /loss/FRt1/assignment_loss‚1.968496142711836 2loss/SRqqt2/assignment_loss‚0.6393284903779406 2loss/FBt1/assignment_loss‚0.004948007426791652 /loss/FRt2/detection_loss‚1.4372459924370051ÔSæúÚ}ú‚(:ò ¦ÎõÒº¤À( disk.nvme0c0n1.out‚36055.97998046875! disk.nvme1c1n1.in‚ 12.11328125( disk.nvme1c1n1.out‚12788.21533203125# disk.dm-0.out‚44321.21044921875 cpu‚26.444239682720095 proc.cpu.threads‚17 network.sent‚ 1.39695e+07 network.recv‚ 1.22415e+06( disk./.usagePercent‚76.4744910019808 disk.nvme0n1.in‚ 129.30859375& disk.nvme1n1.out‚12788.21533203125 disk.md0.in‚0 disk.dm-1.in‚0$ memory_percent‚99.22175407409668% disk./.usageGB‚1174.0746231079102" disk.dm-1.out‚4214.69580078125 disk.rbd0.out‚0" proc.memory.rssMB‚ 2874.1796875 disk.md0.out‚ 145.35546875& disk.nvme0n1.out‚36055.97998046875 disk.nvme1n1.in‚ 12.11328125 disk.dm-0.in‚ 141.421875 disk.rbd0.in‚0' proc.memory.availableMB‚ 127.5078125* proc.memory.percent‚17.542600631713867" disk.nvme0c0n1.in‚ 129.30859375{S¯›û‚(: ¦ÎõÒþÁÇÀ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚111.573' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.190999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Å3ìu2ü- _wandbruntime‚4312  _runtime‚4312¸“--ÆýÀ _step‚853 & lr-AdamW/pg1‚9.590974280233156e-05 & lr-AdamW/pg2‚9.590974280233156e-05 trainer/global_step‚10549 # _timestamp‚1.78450615821391e+09 _runtime‚4312.277072962¬%å´ËþÅ & lr-AdamW/pg1‚9.590974280233156e-05 & lr-AdamW/pg2‚9.590974280233156e-05 trainer/global_step‚10549 # _timestamp‚1.78450615821391e+09 _runtime‚4312.277072962 _step‚853Õ±Du?2ÿ- _wandbruntime‚4312  _runtime‚4312µÝ¢Ý'€×O R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚414295802574848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80327623 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106996350 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7426688376832 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚409003442333516 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106986207 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27122151 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7644504585216 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187323973 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7644682901504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚396275426437120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚414295907636224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚388630921851904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73606878 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚406869219259392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46484724 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7426600354816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 trainer/global_step‚10549 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80327621 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106996350 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚414295802574848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27121901 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚388630946755584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚401599941474288 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 187313828 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845061582148137e+09ÆiÈaâ'ÜO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚414295802574848 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80327623 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106996350 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7426688376832 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚409003442333516 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 106986207 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27122151 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7644504585216 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 187323973 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7644682901504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚396275426437120 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚414295907636224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚388630921851904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚73606878 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚406869219259392 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46484724 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7426600354816 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 trainer/global_step‚10549 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80327621 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106996350 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚414295802574848 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27121901 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚388630946755584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚401599941474288 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 187313828 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845061582148137e+09ÖEU2‚- _wandbruntime‚4312  _runtime‚4312Ô+U ƒáM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107006262 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚406906585626624 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46489017 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187341206 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚414333847051776 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚406906568587264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7427278464512 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80334942 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allo{-ÊÛcated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187331047 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚388666641912832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187341206 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚388666617009152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚414333952120320 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46489014 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚406906585626624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚396312029470208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚396311815771648 _runtime‚4312.538648744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚409040990352832 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73613405 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7645387557376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7427366493696 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27124391 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚855 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107006262 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80334944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚406906568587264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 trainer/global_step‚10549 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚414333952120320 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208xíÀ·ì&„æM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 187341206 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚414333847051776 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚406906568587264 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7427278464512 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80334942 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 187331047 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚388666641912832 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 187341206 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚388666617009152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚414333952120320 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46489014 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚406906585626624 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚396312029470208 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚396311815771648 _runtime‚4312.538648744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚409040990352832 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73613405 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7645387557376 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7427366493696 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27124391 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚855 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107006262 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80334944 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚406906568587264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 trainer/global_step‚10549 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚414333952120320 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107006262 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚406906585626624 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46489017×å„?2…- _wandbruntime‚4312  _runtime‚4312Õ  †… trainer/global_step‚10549 2loss/SRqqt1/assignment_loss‚0.5423029339405916 2loss/SRqqt2/assignment_loss‚0.5334802412968129 /loss/FBt2/detection_loss‚0.4717267520027235 /loss/FRt1/detection_loss‚0.5439915976565844 /loss/FRt2/detection_loss‚1.4110125138336476 1loss/SRqqt2/detection_loss‚0.6008614444062114 .loss/FBt1/detection_loss‚0.564168822268645 % _timestamp‚1.7845061584762473e+09 2loss/FBt1/assignment_loss‚0.002564608592072659 4loss/FBt2/assignment_loss‚0.00017695221526603522 epoch‚13 _runtime‚4312.539122585 _step‚856 /loss/FRt1/assignment_loss‚2.100924991366759 0loss/FRt2/assignment_loss‚0.9343114517769724 1loss/SRqqt1/detection_loss‚1.0277323723012894 %loss/total_loss‚8.733254681657575©I!°‡Š /loss/FRt1/assignment_loss‚2.100924991366759 0loss/FRt2/assignment_loss‚0.9343114517769724 1loss/SRqqt1/detection_loss‚1.0277323723012894 %loss/total_loss‚8.733254681657575 trainer/global_step‚10549 2loss/SRqqt1/assignment_loss‚0.5423029339405916 2loss/SRqqt2/assignment_loss‚0.5334802412968129 /loss/FBt2/detection_loss‚0.4717267520027235 /loss/FRt1/detection_loss‚0.5439915976565844 /loss/FRt2/detection_loss‚1.4110125138336476 1loss/SRqqt2/detection_loss‚0.6008614444062114 .loss/FBt1/detection_loss‚0.564168822268645 % _timestamp‚1.7845061584762473e+09 2loss/FBt1/assignment_loss‚0.002564608592072659 4loss/FBt2/assignment_loss‚0.00017695221526603522 epoch‚13 _runtime‚4312.539122585 _step‚856Ø9h-˜rˆ‚(:ç µÎõÒ«Âý¿ proc.cpu.threads‚17) disk./.usagePercent‚76.47458268889113 disk.dm-1.in‚0$ disk.nvme0c0n1.out‚ 36064.6171875 disk.nvme0n1.in‚ 129.30859375$ memory_percent‚99.22194480895996' proc.memory.availableMB‚ 127.4765625" proc.memory.rssMB‚ 2874.1796875 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 129.30859375 disk.nvme1n1.in‚ 12.11328125 disk.md0.out‚ 145.66015625( disk.nvme1c1n1.out‚12788.74658203125 network.sent‚ 1.4018107e+07% disk./.usageGB‚1174.0760307312012 disk.rbd0.in‚0! disk.nvme1c1n1.in‚ 12.11328125& disk.nvme1n1.out‚12788.74658203125* proc.memory.percent‚17.542600631713867 network.recv‚ 1.2275e+06" disk.dm-1.out‚4214.76220703125" disk.nvme0n1.out‚ 36064.6171875 disk.md0.in‚0 disk.dm-0.in‚ 141.421875 disk.dm-0.out‚ 44329.6484375 cpu‚26.44247301528861®Bªáš‰‚(: µÎõÒ¨œ§À gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚182.564' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚60.85466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Wà@2Š- _wandbruntime‚4331  _runtime‚4331$ëTÈ‹Â & lr-AdamW/pg1‚9.575158611616497e-05 & lr-AdamW/pg2‚9.575158611616497e-05 trainer/global_step‚10599 % _timestamp‚1.7845061775994575e+09 _runtime‚4331.662630908 _step‚857ò+V?ÍŒÇ & lr-AdamW/pg1‚9.575158611616497e-05 & lr-AdamW/pg2‚9.575158611616497e-05 trainer/global_step‚10599 % _timestamp‚1.7845061775994575e+09 _runtime‚4331.662630908 _step‚857Ùt–ŠZ2- _wandbruntime‚4331  _runtime‚4331øÿÝ'Ž×O UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _runtime‚4331.663352499 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10599 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46699227 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7460538096640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7460626118656 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚416198058678272 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80693671 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46699224 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 % _timestamp‚1.7845061776003382e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7679498375168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7679678788608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚416198058678272 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚398095178089472 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107493102 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107482959 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚416198163739648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27245488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚398095383406592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚408737520581632 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7460626118656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107482959 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73944467 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚390415704617984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80693671 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚416198163739648 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚408737537620992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7460538096640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188176630 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚408737537620992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚403443544862288 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚410880772009090 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚408737520581632 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 _step‚858 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚390415679714304 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80693673 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7437244186162 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7437330334636 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚403443527822928 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808=%LŽ¯ ÜO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 188176630 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚408737537620992 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚403443544862288 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚245 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚410880772009090 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚408737520581632 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 _step‚858 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚390415679714304 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚80693673 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7437244186162 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7437330334636 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚403443527822928 VGDeviceStatsMonitor.on_tºœŸ¤3rain_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 _runtime‚4331.663352499 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚10599 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46699227 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7460538096640 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7460626118656 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚416198058678272 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80693671 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46699224 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 % _timestamp‚1.7845061776003382e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7679498375168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7679678788608 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚416198058678272 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚398095178089472 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107493102 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107482959 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚416198163739648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27245488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚398095383406592 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚408737520581632 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7460626118656 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107482959 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚73944467 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚390415704617984 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚80693671 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚416198163739648 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚408737537620992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7460538096640 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680Ú‹¾y¼2- _wandbruntime‚4331  _runtime‚4331NNmç&‘áM M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7437920137606 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7461216206336 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚398131781501440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7680199222784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚410918423216240 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188193849 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27247673 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107492857 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107492857 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188204008 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188204008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7438006286192 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73951456 trainer/global_step‚10599 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚416236208223744 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚408774903988224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46703514 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7680381726208 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73951187 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚390451399775232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚410918320028294 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80700994 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80700994 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚408774903988224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚416236103155200 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73951187 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚390451399775232 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚410918320028294 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚80700994 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚80700994 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚408774903988224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚416236103155200 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7437920137606 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7461216206336 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚398131781501440 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7680199222784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚410918423216240 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188193849 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27247673 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 107492857 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107492857 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188204008 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188204008 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7438006286192 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73951456 trainer/global_step‚10599 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚416236208223744 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚408774903988224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269ÛÿÉàö2“- _wandbruntime‚4331  _runtime‚4331šÇý”þ 1loss/SRqqt1/detection_loss‚1.0179950747261541 /loss/FBt2/detection_loss‚0.6440668002564888 _step‚860 2loss/SRqqt2/assignment_loss‚0.5663605905580991 /loss/FRt2/detection_loss‚1.3936719325406908 .loss/FBt1/detection_loss‚0.956741643926272 epoch‚13 _runtime‚4331.928649643 /loss/FRt2/assignment_loss‚0.931804609986452 1loss/SRqqt1/assignment_loss‚0.702027978046638 0loss/FBt1/assignment_loss‚0.1794976612774574 1loss/FBt2/assignment_loss‚0.03180890854364957 1loss/SRqqt2/detection_loss‚0.6843281646013172 %loss/total_loss‚9.763590553346727 trainer/global_step‚10599 % _timestamp‚1.7845061778658261e+09 /loss/FRt1/assignment_loss‚2.112000949940887 /loss/FRt1/detection_loss‚0.5432862389426215•0ó •ƒ % _timestamp‚1.7845061778658261e+09 /loss/FRt1/assignment_loss‚2.112000949940887 /loss/FRt1/detection_loss‚0.5432862389426215 1loss/SRqqt1/detection_loss‚1.0179950747261541 /loss/FBt2/detection_loss‚0.6440668002564888 _step‚860 2loss/SRqqt2/assignment_loss‚0.5663605905580991 /loss/FRt2/detection_loss‚1.3936719325406908 .loss/FBt1/detection_loss‚0.956741643926272 epoch‚13 _runtime‚4331.928649643 /loss/FRt2/assignment_loss‚0.931804609986452 1loss/SRqqt1/assignment_loss‚0.702027978046638 0loss/FBt1/assignment_loss‚0.1794976612774574 1loss/FBt2/assignment_loss‚0.03180890854364957 1loss/SRqqt2/detection_loss‚0.6843281646013172 %loss/total_loss‚9.763590553346727 trainer/global_step‚10599Ü”.x‘m–‚(:â ÄÎõÒ«žæ¿) disk./.usagePercent‚76.47460554850022" disk.nvme0c0n1.in‚ 129.30859375( disk.nvme0c0n1.out‚36069.47607421875" disk.nvme1c1n1.out‚ 12790.03125 disk.nvme1n1.in‚ 12.11328125" proc.memory.rssMB‚ 2874.1796875 cpu‚26.44083426587333 network.sent‚ 1.4080689e+07 network.recv‚ 1.232142e+06% disk./.usageGB‚1174.0763816833496 disk.dm-0.in‚ 141.421875! disk.nvme1c1n1.in‚ 12.11328125 disk.nvme1n1.out‚ 12790.03125 disk.md0.in‚0' proc.memory.availableMB‚ 127.4453125# disk.dm-0.out‚44334.39794921875 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 129.30859375& disk.nvme0n1.out‚36069.47607421875$ memory_percent‚99.22213554382324 proc.cpu.threads‚17 disk.dm-1.out‚ 4214.9375 disk.rbd0.out‚0 disk.md0.out‚ 146.22265625* proc.memory.percent‚17.542600631713867¹›”&—‚(:… ÄÎõÒ¤ä‹À gpu.0.gpu‚59.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚188.934' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚62.978 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:,Ÿ|˜‚(:ñ ÓÎõÒ‡ôâ¿ disk.dm-1.out‚ 4215.01171875 disk.rbd0.in‚0$ memory_percent‚99.22215938568115( proc.memory.availableMB‚ 127.44140625* proc.memory.percent‚17.542600631713867 network.sent‚ 1.4143007e+07 network.recv‚ 1.236647e+06( disk.nvme0c0n1.out‚36079.52392578125 disk.nvme0n1.in‚ 129.30859375" disk.dm-0.out‚44344.1083984375 disk.dm-1.in‚0 cpu‚26.439262093739455 proc.cpu.threads‚17$ disk./.usageGB‚1174.076503753662! disk.nvme1c1n1.in‚ 12.11328125( disk.nvme1c1n1.out‚12790.70849609375 disk.md0.in‚0 disk.rbd0.out‚0& disk.nvme0n1.out‚36079.52392578125 disk.nvme1n1.in‚ 12.11328125& disk.nvme1n1.out‚12790.70849609375 disk.md0.out‚ 146.6640625 disk.dm-0.in‚ 141.421875" proc.memory.rssMB‚ 2874.1796875) disk./.usagePercent‚76.47461349966859" disk.nvme0c0n1.in‚ 129.30859375´â¿›™‚(: ÓÎõÒìÀÀ gpu.0.gpu‚78.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚48.0 gpu.0.powerWatts‚186.296' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚62.09866666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²)8x2š- _wandbruntime‚4351  _runtime‚4351Lzț & lr-AdamW/pg1‚9.559279374487308e-05 & lr-AdamW/pg2‚9.559279374487308e-05 trainer/global_step‚10649 % _timestamp‚1.7845061969726033e+09 _runtime‚4351.037627842 _step‚861ÛäWëÍœÇ & lr-AdamW/pg1‚9.559279374487308e-05 & lr-AdamW/pg2‚9.559279374487308e-05 trainer/global_step‚10649 % _timestamp‚1.7845061969726033e+09 _runtime‚4351.037627842 _step‚861ÝIèRb2- _wandbruntime‚4351  _runtime‚4351¡Pƒ žÖO NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚399915083105280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚399914873593856 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74282482 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27368758 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7714436017152 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚418100419843072 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚418100314781696 F8DevgIŒðÐ#iceStatsMonitor.on_train_batch_start/active.all.freed‚ 189039432 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46913724 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7714620624896 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚410605855982592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7494475838464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚405287131210928 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81059723 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81059723 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7494475838464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚412758204872498 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚418100419843072 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚392200462480384 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚410605838943232 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚412758308060332 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189049577 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107989854 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107989854 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 _runtime‚4351.037691983 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376&lÀá'ŸÛO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81059723 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7494475838464 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚412758204872498 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚418100419843072 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚392200462480384 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚410605838943232 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚412758308060332 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189049577 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107989854 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 107989854 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 _runtime‚4351.037691983 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚399915083105280 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚399914873593856 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74282482 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27368758 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7714436017152 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚418100419843072 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚418100314781696 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189039432 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46913724 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7714620624896 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚410605855982592 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7494475838464 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚405287131210928 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81059723 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152Þ ~¯n2 - _wandbruntime‚4351  _runtime‚4351ó#âMç&¡áM A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27371524 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399951265558016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7471835761600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81067044 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81067044 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚405324003278688 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7715132824064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚412795752891702 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚410643222349824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7495153948160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189066810 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7495241977344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74289284 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189066810 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 % _timestamp‚1.7845061972383125e+09 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46918014 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74289541 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚410643222349824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7471749613014 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚410643205310464 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7495153948160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7495241977344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586z(ô ¢æM A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27371524 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚399951265558016 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7471835761600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81067044 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81067044 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚405324003278688 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7715132824064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚412795752891702 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚257 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚410643222349824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7495153948160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189066810 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7495241977344 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74289284 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189066810 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 % _timestamp‚1.7845061972383125e+09 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46918014 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74289541 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚410643222349824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7471749613014 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚410643205310464 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7495153948160 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7495241977344 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586ßy 6$2£- _wandbruntime‚4351  _runtime‚4351Ö‰‚b ¤… trainer/global_step‚10649 0loss/FRt1/assignment_loss‚2.0252750440190237 2loss/SRqqt1/assignment_loss‚0.5979918157635338 2loss/SRqqt2/assignment_loss‚0.6671295324291956 epoch‚13 % _timestamp‚1.7845061972393236e+09 3loss/FBt2/assignment_loss‚0.0014710217347799948 1loss/SRqqt2/detection_loss‚0.6691378229810714 /loss/FBt1/detection_loss‚0.9920859468169511 /loss/FBt2/detection_loss‚0.4977790062607787 %loss/total_loss‚9.993854541756876 _runtime‚4351.302304949 1loss/FBt1/assignment_loss‚0.33385450001232114 /loss/FRt1/detection_loss‚0.5934862072252448 /loss/FRt2/detection_loss‚1.5542555676967216 1loss/SRqqt1/detection_loss‚1.0826139653476679 _step‚864 0loss/FRt2/assignment_loss‚0.97877411146958672œî饊 0loss/FRt1/assignment_loss‚2.0252750440190237 2loss/SRqqt1/assignment_loss‚0.5979918157635338 2loss/SRqqt2/assignment_loss‚0.6671295324291956 epoch‚13 % _timestamp‚1.7845061972393236e+09 3loss/FBt2/assignment_loss‚0.0014710217347799948 1loss/SRqqt2/detection_loss‚0.6691378229810714 /loss/FBt1/detection_loss‚0.9920859468169511 /loss/FBt2/detection_loss‚0.4977790062607787 %loss/total_loss‚9.993854541756876 _runtime‚4351.302304949 1loss/FBt1/assignment_loss‚0.33385450001232114 /loss/FRt1/detection_loss‚0.5934862072252448 /loss/FRt2/detection_loss‚1.5542555676967216 1loss/SRqqt1/detection_loss‚1.0826139653476679 _step‚864 0loss/FRt2/assignment_loss‚0.9787741114695867 trainer/global_step‚10649à ?v¦‚(:ë âÎõÒÎöõ¿ disk.md0.out‚ 146.921875 disk.dm-1.in‚0" disk.dm-1.out‚4215.20947265625 disk.nvme1n1.in‚ 12.11328125# proc.memory.availableMB‚126.875* proc.memory.percent‚17.542600631713867 proc.cpu.threads‚17 network.sent‚ 1.4147957e+07 disk.dm-0.in‚ 141.421875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 129.30859375% disk.nvme0n1.out‚36115.4638671875( disk.nvme1c1n1.out‚12792.01904296875" proc.memory.rssMB‚ 2874.1796875( disk./.usagePercent‚76.4591477317543% disk./.usageGB‚1173.8390655517578" disk.nvme0c0n1.in‚ 129.30859375' disk.nvme0c0n1.out‚36115.4638671875 network.recv‚ 1.240176e+06 disk.md0.in‚0! disk.dm-0.out‚44380.607421875! disk.nvme1c1n1.in‚ 12.11328125& disk.nvme1n1.out‚12792.01904296875$ memory_percent‚99.22561645507812 cpu‚26.437641360784603\ÉÐŽ§‚(:ƒ âÎõÒº±‡À gpu.0.gpu‚54.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚121.14' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚40.38 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<{Ú 2¨- _wandbruntime‚4370  _runtime‚43704ho:È©Â & lr-AdamW/pg1‚9.543336823183797e-05 & lr-AdamW/pg2‚9.543336823183797e-05 trainer/global_step‚10699 % _timestamp‚1.7845062159413993e+09 _runtime‚4370.005660456 _step‚865C}8껂 & lr-AdamW/pg1‚9.543336823183797e-05 & lr-AdamW/pg2‚9.543336823183797e-05 trainer/global_step‚10699 % _timestamp‚1.7845062159413993e+09 _runtime‚4370.005660456 _step‚865áH CA2«- _wandbruntime‚4370  _runtime‚4370ëy0—Ý'¬×O \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚407130734598928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74620688 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189902234 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7528501602304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚393985195439104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚412474157304832 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚412474174344192 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚420002570885120 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108476463 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚420002570885120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47128227 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚10699 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7749569679872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _runtime‚4370.005724806 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚412474157304832 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7749391363584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚420002675946496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚393985220342784 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7845062159420881e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81425771 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚414635637735906 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _step‚866 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7504903136978 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108476463 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚420002675946496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81425773 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27492218 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚414635740923740 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7528413580288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108486606 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚401734790022656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81425773 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108486606 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚401734586802688ÍnÄìâ'­ÜO M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74620442 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7845062159420881e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81425771 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚414635637735906 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _step‚866 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7504903136978 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108476463 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚420002675946496 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81425773 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27492218 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚414635740923740 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7528413580288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚243 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108486606 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚401734790022656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81425773 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108486606 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚401734586802688 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚407130734598928 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74620688 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 189902234 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7528501602304 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚393985195439104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚412474157304832 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚412474174344192 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚420002570885120 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108476463 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚420002570885120 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47128227 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚10699 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7749569679872 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _runtime‚4370.005724806 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚412474157304832 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7749391363584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚420002675946496 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚393985220342784 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189929612 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7505579088422 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚407167606666688 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108496518 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7529179719168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚412511523672064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚414673288943056 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7529091689984 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74627062 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚412511523672064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81433092 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189929612 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚401771185702912 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚401770972004352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7750270202880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108496518 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚414673185755110 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47132517 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81433094 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚394020915500032 _runtime‚4370.587407071 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81433092 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27494548 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚407167623706048 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7529179719168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚420040720430592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47132514 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189919453 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚867 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚420040720430592 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚10699 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330Øì&°æM SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108496518 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚414673185755110 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47132517 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81433094 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚245 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚394020915500032 _runtime‚4370.587407071 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81433092 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27494548 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚407167623706048 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7529179719168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚420040720430592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47132514 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189919453 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚867 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚420040720430592 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚10699 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚412511540711424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7845062165240762e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚420040615362048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚412511540711424 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7750081408000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81433094 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108486361 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27494793 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108486361 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189929612 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7505579088422 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚407167606666688 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108496518 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7529179719168 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚412511523672064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚414673288943056 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7529091689984 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74627062 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚412511523672064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81433092 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189929612 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚401771185702912 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚401770972004352 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7750270202880 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816ãLåÚ½2±- _wandbruntime‚4370  _runtime‚43708øZy ²„ 1loss/SRqqt1/detection_loss‚1.0291757382825018 /loss/FBt2/detection_loss‚0.7581246512749329 epoch‚13 /loss/FRt1/detection_loss‚0.5804665950601144 $ _timestamp‚1.784506216525139e+09 _step‚868 2loss/SRqqt1/assignment_loss‚0.5787512105187246 .loss/FRt2/detection_loss‚1.576786873738826 trainer/global_step‚10699 1loss/SRqqt2/detection_loss‚0.6968316772127716 /loss/FBt1/detection_loss‚0.8408215084396626 %loss/total_loss‚9.559574726312356 _runtime‚4370.588222661 0loss/FRt1/assignment_loss‚1.9595194135894294 0loss/FRt2/assignment_loss‚0.9358659497762131 2loss/SRqqt2/assignment_loss‚0.6006076135718028 3loss/FBt1/assignment_loss‚0.0008331652571126577 2loss/FBt2/assignment_loss‚0.001790329590265498¤¿S³‰ 0loss/FRt1/assignment_loss‚1.9595194135894294 0loss/FRt2/assignment_loss‚0.9358659497762131 2loss/SRqqt2/assignment_loss‚0.6006076135718028 3loss/FBt1/assignment_loss‚0.0008331652571126577 2loss/FBt2/assignment_loss‚0.001790329590265498 1loss/SRqqt1/detection_loss‚1.0291757382825018 /loss/FBt2/detection_loss‚0.7581246512749329 epoch‚13 /loss/FRt1/detection_loss‚0.5804665950601144 $ _timestamp‚1.784506216525139e+09 _step‚868 2loss/SRqqt1/assignment_loss‚0.5787512105187246 .loss/FRt2/detection_loss‚1.576786873738826 trainer/global_step‚10699 1loss/SRqqt2/detection_loss‚0.6968316772127716 /loss/FBt1/detection_loss‚0.8408215084396626 %loss/total_loss‚9.559574726312356 _runtime‚4370.588222661äû¾^Öw´‚(:ì ñÎõÒþ”õ¿" proc.memory.rssMB‚ 2874.1796875 network.recv‚ 1.243937e+06% disk./.usageGB‚1173.8390426635742 disk.rbd0.out‚0 disk.nvme0n1.in‚ 129.30859375! disk.nvme1c1n1.in‚ 12.11328125( disk.nvme1c1n1.out‚12792.72998046875 disk.dm-0.in‚ 141.421875 disk.dm-0.out‚ 44388.59375 disk.rbd0.in‚0% disk.nvme0n1.out‚36123.8369140625% proc.memory.availableMB‚ 127.09375" disk.nvme0c0n1.in‚ 129.30859375' disk.nvme0c0n1.out‚36123.8369140625 disk.md0.in‚0$ memory_percent‚99.22428131103516* proc.memory.percent‚17.542600631713867 cpu‚26.436090604392803 proc.cpu.threads‚17 network.sent‚ 1.4210294e+07) disk./.usagePercent‚76.45914624091023 disk.nvme1n1.in‚ 12.11328125& disk.nvme1n1.out‚12792.72998046875 disk.md0.out‚ 147.39453125 disk.dm-1.in‚0" disk.dm-1.out‚4215.29150390625Š€"Cœµ‚(:‘ ñÎõÒЗÚÀ gpu.0.gpu‚71.0 gpu.0.memory‚10+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚112.001' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.33366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â]í2¶- _wandbruntime‚4389  _runtime‚4389³xÞvÇ·Á & lr-AdamW/pg1‚9.527331213058288e-05 & lr-AdamW/pg2‚9.527331213058288e-05 trainer/global_step‚10749 $ _timestamp‚1.784506235235878e+09 _runtime‚4389.299053956 _step‚869EÓIÌ¸Æ & lr-AdamW/pg1‚9.527331213058288e-05 & lr-AdamW/pg2‚9.527331213058288e-05 trainer/global_step‚10749 $ _timestamp‚1.784506235235878e+09 _runtime‚4389.299053956 _step‚869åãNÐV2¹- _wandbruntime‚4389  _runtime‚4389ê}<æÜ'ºÖO GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81791823 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚414342475666432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27615077 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108973215 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7784498733568 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚421904826988544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81791821 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚395769978205184 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚395769953301504 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚421904932049920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚421904826988544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108973215 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚403554476938752 trainer/global_step‚10749 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108983358 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚414342475666432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7538732612386 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚416513173787148 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚408974355026288 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7562351322112 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7784320417280 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7562439344128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 190775181 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7538818760860 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _runtime‚4389.299867117 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7562351322112 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74957801 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27615309 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81791821 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47342727 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚414342492705792 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81791823 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7562439344128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚403554273718784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74958036 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚408974337986928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464*%e!+»ÛO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7562351322112 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7784320417280 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 JBDeviceStatsMonitor.on_train_batch-§x|¶"_start/segment.large_pool.current‚330 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7562439344128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 190775181 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7538818760860 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 _runtime‚4389.299867117 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7562351322112 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74957801 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27615309 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81791821 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47342727 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚414342492705792 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81791823 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7562439344128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚403554273718784 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74958036 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚408974337986928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81791823 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚414342475666432 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27615077 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 178316288 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108973215 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7784498733568 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚421904826988544 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81791821 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚395769978205184 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚395769953301504 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚421904932049920 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚421904826988544 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108973215 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚403554476938752 trainer/global_step‚10749 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 108983358 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚414342475666432 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7538732612386 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚416513173787148 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚408974355026288æ×{‰2¼- _wandbruntime‚4389  _runtime‚4389䟹cç&½áM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 190792414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚421942976534016 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108983113 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7539494712416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81799142 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚421942871465472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81799144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚414379859073024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚234 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚409011227094048 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚414379842033664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190782255 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚409011210054688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108983113 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚871 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74964766 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚416550721806464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7563117460992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47347014 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚421942976534016 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108993270 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚414379859073024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47347017 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7563029431808 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚403590665056256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74964532 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7845062354990127e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81799144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚416550618618518 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190792414 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81799142 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚395805648458752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 _runtime‚4389.562350229 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚414379842033664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7563029431808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108993270 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚395805673362432 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7539408563830 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7785016597504 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482¼g|êì&¾æM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81799144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚416550618618518 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 190792414 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81799142 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚395805648458752 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 _runtime‚4389.562350229 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚414379842033664 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7563029431808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 108993270 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚395805673362432 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7539408563830 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7785016597504 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 190792414 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚421942976534016 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108983113 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7539494712416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81799142 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚421942871465472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81799144 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚414379859073024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚234 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚409011227094048 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚414379842033664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 190782255 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚409011210054688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108983113 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚871 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74964766 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚416550721806464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7563117460992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47347014 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚421942976534016 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 108993270 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚414379859073024 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47347017 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7563029431808 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚231 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚403590665056256 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74964532 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7845062354990127e+09ç  âÃ2¿- _wandbruntime‚4389  _runtime‚4389Ϧw À† 2loss/FBt1/assignment_loss‚0.004174369255206248 1loss/SRqqt1/detection_loss‚1.0935475016791683 /loss/FBt1/detection_loss‚1.4806979734982764 1loss/SRqqt2/assignment_loss‚0.625066932315369 1loss/SRqqt2/detection_loss‚0.6225444914126355 3loss/FBt2/assignment_loss‚0.0016547737840808176 /loss/FRt1/detection_loss‚0.5594800347545095 /loss/FRt2/detection_loss‚1.5011797354742884 %loss/total_loss‚9.929813232163374 epoch‚13 trainer/global_step‚10749 _step‚872 0loss/FRt1/assignment_loss‚2.1832048195526315 0loss/FRt2/assignment_loss‚0.9639088494777679 3loss/SRqqt1/assignment_loss‚0.49736986006097117 0loss/FBt2/detection_loss‚0.39698389089847014 % _timestamp‚1.7845062355003457e+09 _runtime‚ 4389.56319536âºrÁ‹ 1loss/SRqqt2/detection_loss‚0.6225444914126355 3loss/FBt2/assignment_loss‚0.0016547737840808176 /loss/FRt1/detection_loss‚0.5594800347545095 /loss/FRt2/detection_loss‚1.5011797354742884 %loss/total_loss‚9.929813232163374 epoch‚13 trainer/global_step‚10749 _step‚872 0loss/FRt1/assignment_loss‚2.1832048195526315 0loss/FRt2/assignment_loss‚0.9639088494777679 3loss/SRqqt1/assignment_loss‚0.49736986006097117 0loss/FBt2/detection_loss‚0.39698389089847014 % _timestamp‚1.7845062355003457e+09 _runtime‚ 4389.56319536 2loss/FBt1/assignment_loss‚0.004174369255206248 1loss/SRqqt1/detection_loss‚1.0935475016791683 /loss/FBt1/detection_loss‚1.4806979734982764 1loss/SRqqt2/assignment_loss‚0.625066932315369èYžöDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚405373976493056 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7819447839744 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚418390606650556 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7596289063936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚416210811067392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚423807188153344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚423807083091968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7845062545521574e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚416210811067392 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 trainer/global_step‚10799 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7596377085952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82157871 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7572562087794 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75295243 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7596377085952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27738016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚418390503462722 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75294986 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚405374183907328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚410817941374928 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 _runtime‚4408.615167271 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109480110 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27737762 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423807083091968 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109469967 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47557224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚410817958414288 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82157873 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚423807188153344 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82157873 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82157871 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚416210794028032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191637983 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 _step‚874o3t4â'ÉÜO E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109480110 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7596289063936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47557227 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚397554711163904 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109469967 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47557224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚410817958414288 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82157873 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚423807188153344 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82157873 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82157871 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚416210794028032 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191637983 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 _step‚874 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚397554736067584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7572648236268 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 191627838 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚405373976493056 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7819447839744 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚418390606650556 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7596289063936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚416210811067392 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚423807188153344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚423807083091968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7845062545521574e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚416210811067392 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 trainer/global_step‚10799 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7596377085952 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82157871 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚257 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚254 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7572562087794 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75295243 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7596377085952 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27738016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚418390503462722 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75294986 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚405374183907328 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚410817941374928 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 _runtime‚4408.615167271 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109480110 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27737762 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚423807083091968 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚423845232637440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚405410575770112 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚418428051481926 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 $ _timestamp‚1.784506254811296e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82165192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75301743 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27740464 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚410854813442688 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚410854830482048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚416248177434624 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚416248160395264 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚10799 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109479865 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109479865 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚405410370460160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82165192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚397590406321152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191645057 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7573238039238 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109490022 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚238 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191655216 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191645057 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7820144545280 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7596967173632 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7597055202816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75301981 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109490022 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27740229 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82165194 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0(?Z ÌåM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191655216 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191645057 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7820144545280 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7596967173632 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7597055202816 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75301981 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109490022 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27740229 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82165194 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 PÏÒ"¶[BDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚423845232637440 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚405410575770112 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚418428051481926 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 $ _timestamp‚1.784506254811296e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82165192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75301743 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27740464 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚410854813442688 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚410854830482048 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚416248177434624 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚416248160395264 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚10799 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109479865 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109479865 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚405410370460160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82165192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚397590406321152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 191645057 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7573238039238 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109490022 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚238ëöA#ä2Í- _wandbruntime‚4408  _runtime‚4408³É£q ΃ 0loss/FRt2/assignment_loss‚0.7685430231952268 /loss/FRt1/detection_loss‚0.5553984558971228 /loss/FRt2/detection_loss‚1.5664271374688987 % _timestamp‚1.7845062548122725e+09 2loss/SRqqt1/assignment_loss‚0.5418245743296651 1loss/FBt1/assignment_loss‚0.04413026993396376 1loss/SRqqt2/detection_loss‚0.6626717620164145 $loss/total_loss‚9.32753875276368 trainer/global_step‚10799 _step‚876 epoch‚13 2loss/SRqqt2/assignment_loss‚0.6388388686000918 3loss/FBt2/assignment_loss‚0.0030196214047781248 1loss/SRqqt1/detection_loss‚0.9463119178255593 /loss/FBt1/detection_loss‚1.1734569973367102 /loss/FBt2/detection_loss‚0.4959293304732694 _runtime‚4408.875189794 /loss/FRt1/assignment_loss‚1.930986794281979w$HSψ 0loss/FRt2/assignment_loss‚0.7685430231952268 /loss/FRt1/detection_loss‚0.5553984558971228 /loss/FRt2/detection_loss‚1.5664271374688987 % _timestamp‚1.7845062548122725e+09 2loss/SRqqt1/assignment_loss‚0.5418245743296651 1loss/FBt1/assignment_loss‚0.04413026993396376 1loss/SRqqt2/detection_loss‚0.6626717620164145 $loss/total_loss‚9.32753875276368 trainer/global_step‚10799 _step‚876 epoch‚13 2loss/SRqqt2/assignment_loss‚0.6388388686000918 3loss/FBt2/assignment_loss‚0.0030196214047781248 1loss/SRqqt1/detection_loss‚0.9463119178255593 /loss/FBt1/detection_loss‚1.1734569973367102 /loss/FBt2/detection_loss‚0.4959293304732694 _runtime‚4408.875189794 /loss/FRt1/assignment_loss‚1.930986794281979ì`ÁG:šÐ‚(: ÏõÒßÀÁ¿ gpu.0.gpu‚9.0 gpu.0.memory‚1+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚201.704' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚67.23466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®©h½wÑ‚(:ì ÏõÒüŠÖ¿ network.sent‚ 1.4334994e+07 disk.nvme1n1.in‚ 13.57421875% disk.nvme1n1.out‚12794.1826171875 disk.nvme0n1.in‚ 135.26953125$ memory_percent‚99.22473430633545% disk./.usageGB‚1173.8389930725098' disk.nvme1c1n1.out‚12794.1826171875 disk.dm-0.in‚ 148.84375 disk.dm-0.out‚44395.11328125 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 135.26953125" proc.memory.rssMB‚ 2874.1796875 network.recv‚ 1.252276e+06) disk./.usagePercent‚76.45914301074808 disk.md0.in‚0 disk.rbd0.in‚0& disk.nvme0c0n1.out‚36130.568359375( proc.memory.availableMB‚ 127.01953125* proc.memory.percent‚17.542600631713867 cpu‚26.432966921475426! disk.nvme1c1n1.in‚ 13.57421875 disk.md0.out‚ 148.01171875 disk.dm-1.in‚0! disk.dm-1.out‚4215.6025390625$ disk.nvme0n1.out‚36130.568359375 proc.cpu.threads‚17 ³oÒ‚(:ä žÏõÒÆ‹á¿ disk.md0.out‚ 148.390625 disk.dm-0.in‚ 149.203125 disk.dm-1.in‚0" disk.dm-1.out‚4215.77587890625 disk.nvme1c1n1.in‚ 13.578125( disk.nvme1c1n1.out‚12882.66455078125 disk.md0.in‚0 proc.cpu.threads‚17 network.sent‚ 1.4397419e+07 network.recv‚ 1.257045e+06) disk./.usagePercent‚76.47464058333587$ disk./.usageGB‚1174.076919555664 disk.rbd0.in‚0& proc.memory.availableMB‚ 126.953125* proc.memory.percent‚17.542600631713867 cpu‚26.431333876625832( disk.nvme0c0n1.out‚36314.64404296875 disk.nvme0n1.in‚135.625& disk.nvme0n1.out‚36314.64404296875& disk.nvme1n1.out‚12882.66455078125# disk.dm-0.out‚44666.66162109375 disk.rbd0.out‚0$ memory_percent‚99.22513961791992" proc.memory.rssMB‚ 2874.1796875 disk.nvme0c0n1.in‚135.625 disk.nvme1n1.in‚ 13.578125KMn;›Ó‚(: žÏõÒŠÔà¿ gpu.0.gpu‚69.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚173.279' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚57.75966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0y¼Ñ„2Ô- _wandbruntime‚4428  _runtime‚4428DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚418079112389632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚425709339195392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82523921 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚425709444256768 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47771724 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75632737 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚399339469026304 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109966719 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 190899200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82523921 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7854127672320 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7630314827776 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚420268039513964 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7630226805760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75632476 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚420267936326130 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚418079129428992 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109976862 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82523923 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7606391563202 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7630226805760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27861010 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚425709444256768 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 _runtime‚4428.057585416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚425709339195392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27860752 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215802880 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109966719 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚418079129428992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109976862 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚407193596698624 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 _step‚878 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82523923 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 192490640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚407193812501504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7854318571520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7606477711676 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚418079112389632 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚425709339195392 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82523921 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚425709444256768 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47771724 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75632737 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚258 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚399339469026304 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109966719 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 190899200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82523921 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7854127672320 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7630314827776 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚420268039513964 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7630226805760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75632476 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚420267936326130 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚418079129428992 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109976862 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82523923 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7606391563202 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7630226805760 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27861010 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚425709444256768 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 _runtime‚4428.057585416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚425709339195392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27860752 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 215802880 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 109966719 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚418079129428992 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109976862 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚407193596698624 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 _step‚878 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚82523923 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚261 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 192490640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚407193812501504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7854318571520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7606477711676 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚418116495796224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚418116478756864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 192518018 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7607067514646 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 192518018 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82531244 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7630992944640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7607153663232 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚412698416830688 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7630904915456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27863195 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109986774 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7630904915456 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚418116495796224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109976617 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75639209 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚420305587533280 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82531244 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7854823769600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10849 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚399375189087232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845062742538662e+09bDHì&ÜæM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚425747488740864 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82531242 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚418116478756864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7630992944640 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚418116495796224 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚418116478756864 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 192989184 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 192518018 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7607067514646 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 192518018 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82531244 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7630992944640 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7607153663232 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚412698416830688 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7630904915456 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27863195 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 109986774 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7630904915456 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚418116495796224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109976617 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75639209 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚420305587533280 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82531244 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 217892864 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7854823769600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10849 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚399375189087232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7845062742538662e+09 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75639467 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚255 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 109976617 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚399375164183552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47776014 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 192507859 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚425747488740864 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47776017 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚412698433870048 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0ïtþnª2Ý- _wandbruntime‚4428  _runtime‚4428”I. Þ„ /loss/FRt1/detection_loss‚0.5909899968376986 /loss/FBt1/detection_loss‚0.6690709366989525 0loss/FRt1/assignment_loss‚2.1220955809768367 2loss/FBt1/assignment_loss‚0.009246674802689678 1loss/SRqqt2/detection_loss‚0.6519257827745178 epoch‚13 trainer/global_step‚10849 /loss/FRt2/assignment_loss‚0.881227667254161 1loss/SRqqt2/assignment_loss‚0.537595509849983 3loss/FBt2/assignment_loss‚5.890773616440671e-05 1loss/SRqqt1/detection_loss‚0.9952246379341532 /loss/FBt2/detection_loss‚0.5194394538752162 %loss/total_loss‚9.087098970427373 _step‚880 2loss/SRqqt1/assignment_loss‚0.6237063993825782 /loss/FRt2/detection_loss‚1.4865174223044206 % _timestamp‚1.7845062742548826e+09 _runtime‚4428.317803633 Âg߉ 1loss/SRqqt1/detection_loss‚0.9952246379341532 /loss/FBt2/detection_loss‚0.5194394538752162 %loss/total_loss‚9.087098970427373 _step‚880 2loss/SRqqt1/assignment_loss‚0.6237063993825782 /loss/FRt2/detection_loss‚1.4865174223044206 % _timestamp‚1.7845062742548826e+09 _runtime‚4428.317803633 /loss/FRt1/detection_loss‚0.5909899968376986 /loss/FBt1/detection_loss‚0.6690709366989525 0loss/FRt1/assignment_loss‚2.1220955809768367 2loss/FBt1/assignment_loss‚0.009246674802689678 1loss/SRqqt2/detection_loss‚0.6519257827745178 epoch‚13 trainer/global_step‚10849 /loss/FRt2/assignment_loss‚0.881227667254161 1loss/SRqqt2/assignment_loss‚0.537595509849983 3loss/FBt2/assignment_loss‚5.890773616440671e-05ðƒù[Bnà‚(:ã ­ÏõÒÁ¼ŽÃ% disk.nvme0n1.out‚36334.2705078125( disk.nvme1c1n1.out‚12883.33642578125 disk.md0.out‚ 148.80078125 disk.dm-0.in‚ 149.203125$ memory_percent‚99.22535419464111 network.recv‚ 1.26034e+06 disk.md0.in‚0" disk.dm-0.out‚44685.9716796875" disk.dm-1.out‚4215.87353515625( proc.memory.availableMB‚ 126.91796875 cpu‚26.42978389395001 disk.nvme0n1.in‚135.625 disk.nvme1c1n1.in‚ 13.578125 disk.nvme1n1.in‚ 13.578125 disk.dm-1.in‚0" proc.memory.rssMB‚ 2874.1796875* proc.memory.percent‚17.542600631713867 network.sent‚ 1.440229e+07) disk./.usagePercent‚76.47463809859576' disk.nvme0c0n1.out‚36334.2705078125& disk.nvme1n1.out‚12883.33642578125 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17% disk./.usageGB‚1174.0768814086914 disk.nvme0c0n1.in‚135.625^‘C›á‚(: ­ÏõÒßá¼Â gpu.0.gpu‚59.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚115.748' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚38.58266666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§ÿ/2â- _wandbruntime‚4447  _runtime‚4447K¶ÅÈã trainer/global_step‚10899 % _timestamp‚1.7845062934716837e+09 _runtime‚4447.534880408 _step‚881 & lr-AdamW/pg1‚9.478938598398522e-05 & lr-AdamW/pg2‚9.478938598398522e-05n=TÍäÇ & lr-AdamW/pg1‚9.478938598398522e-05 & lr-AdamW/pg2‚9.478938598398522e-05 trainer/global_step‚10899 % _timestamp‚1.7845062934716837e+09 _runtime‚4447.534880408 _step‚881ñ\>52å- _wandbruntime‚4447  _runtime‚4447X*€ˆs%æ×O H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47986227 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚419947430751232 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110473614 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75969795 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7664252569600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110463471 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47986224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27983568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚409013433093632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7888988304896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚427611700360192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7845062934727366e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75969563 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚401124226888704 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7640221038610 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82889971 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7889181301248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192996352 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚427611595298816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚419947430751232 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚427611700360192 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚427611595298816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚229 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7664252569600 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚422145472377372 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚414505165190288 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚414505148150928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 trainer/global_step‚10899 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚232 ZLDeviceStatsMonitor.on_train_batch_starþdz/jt/inactive_split_bytes.large_pool.peak‚ 743477248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7640307187084 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47986227 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚419947430751232 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110473614 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75969795 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7664252569600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110463471 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47986224 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27983568 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚409013433093632 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7888988304896 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚427611700360192 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7845062934727366e+09 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75969563 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚401124226888704 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7640221038610 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82889971 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7889181301248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192996352 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚427611595298816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚419947430751232 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚427611700360192 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚427611595298816 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016òÞÊ3K2è- _wandbruntime‚4447  _runtime‚4447Àkíæ&éàM 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚414542037258048 _runtime‚4447.797084442 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110483526 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7664930686464 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7664842657280 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193370661 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚224 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7664930686464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27985774 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193380820 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110483526 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47990517 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7640983138640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75976288 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193380820 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47990514 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚419984814157824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193370661 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27985998 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚414542020218688 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚401159922045952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110473369 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75976515 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚227 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚10899 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚409049818038272 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚419984814157824 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚401159946949632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82897292 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 _step‚883 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚422183020396688 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚422182917208742 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7664842657280 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7640896990054 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82897292 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚427649744844288 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚427649744844288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚409049610631168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚427649639775744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7889688585216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82897294 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚419984797118464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184¿[=ë&êåM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚427649744844288 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚427649744844288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚409049610631168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚427649639775744 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7889688585216 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82897294 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚419984797118464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚414542037258048 _runtime‚4447.797084442 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110483526 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7664930686464 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7664842657280 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193370661 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚224 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7664930686464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27985774 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193380820 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110483526 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47990517 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7640983138640 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75976288 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193380820 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47990514 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚419984814157824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 193370661 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27985998 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚414542020218688 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚401159922045952 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110473369 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75976515 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚227 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚10899 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚409049818038272 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚419984814157824 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚401159946949632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82897292 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 _step‚883 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚422183020396688 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚422182917208742 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7664842657280 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7640896990054 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82897292 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816󪽪2ë- _wandbruntime‚4447  _runtime‚4447ˆñ\ìþ /loss/FRt2/assignment_loss‚0.856637564022094 2loss/SRqqt1/assignment_loss‚0.5778972032744667 1loss/FBt1/assignment_loss‚0.01040558328254736 /loss/FBt1/detection_loss‚1.1137196346299745 /loss/FRt1/assignment_loss‚2.007768017905099 0loss/SRqqt1/detection_loss‚0.994958319034657 1loss/SRqqt2/detection_loss‚0.6149320108394584 /loss/FBt2/detection_loss‚0.4409263559927543 1loss/FBt2/assignment_loss‚0.01835828394629895 /loss/FRt1/detection_loss‚0.4639727638805383 .loss/FRt2/detection_loss‚1.346617907172913 epoch‚13 trainer/global_step‚10899 2loss/SRqqt2/assignment_loss‚0.6059629917998027 %loss/total_loss‚9.052156635780605 $ _timestamp‚1.784506293734701e+09 _runtime‚4447.797585762 _step‚884þ¢Wæ íƒ 1loss/FBt2/assignment_loss‚0.01835828394629895 /loss/FRt1/detection_loss‚0.4639727638805383 .loss/FRt2/detection_loss‚1.346617907172913 epoch‚13 trainer/global_step‚10899 2loss/SRqqt2/assignment_loss‚0.6059629917998027 %loss/total_loss‚9.052156635780605 $ _timestamp‚1.784506293734701e+09 _runtime‚4447.797585762 _step‚884 /loss/FRt2/assignment_loss‚0.856637564022094 2loss/SRqqt1/assignment_loss‚0.5778972032744667 1loss/FBt1/assignment_loss‚0.01040558328254736 /loss/FBt1/detection_loss‚1.1137196346299745 /loss/FRt1/assignment_loss‚2.007768017905099 0loss/SRqqt1/detection_loss‚0.994958319034657 1loss/SRqqt2/detection_loss‚0.6149320108394584 /loss/FBt2/detection_loss‚0.4409263559927543ôÚ?3î‚(:â ¼ÏõÒÍÎó¿) disk./.usagePercent‚76.47458865226741 disk.nvme0n1.in‚135.625 disk.nvme1c1n1.in‚ 13.578125 disk.md0.in‚0# disk.dm-0.out‚44689.78662109375( proc.memory.availableMB‚ 126.88671875 cpu‚26.42860441635984 network.sent‚ 1.4464814e+07( disk.nvme0c0n1.oÚ×Ë:ut‚36338.16455078125& disk.nvme0n1.out‚36338.16455078125 disk.nvme1n1.in‚ 13.578125 disk.md0.out‚ 149.33984375 disk.dm-0.in‚ 149.203125 disk.rbd0.in‚0" proc.memory.rssMB‚ 2874.1796875 network.recv‚ 1.264895e+06% disk./.usageGB‚1174.0761222839355& disk.nvme1c1n1.out‚12884.615234375$ disk.nvme1n1.out‚12884.615234375 disk.rbd0.out‚0 disk.nvme0c0n1.in‚135.625 disk.dm-1.in‚0! disk.dm-1.out‚4216.0888671875# memory_percent‚99.2255449295044* proc.memory.percent‚17.542600631713867 proc.cpu.threads‚17É×'›ï‚(: ¼ÏõÒÝÀÔÀ gpu.0.gpu‚60.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚198.076' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.02533333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’ù¶2ð- _wandbruntime‚4466  _runtime‚4466‹ŽÙËÈñ & lr-AdamW/pg1‚9.462683326646811e-05 & lr-AdamW/pg2‚9.462683326646811e-05 trainer/global_step‚10949 % _timestamp‚1.7845063124400125e+09 _runtime‚4466.504612948 _step‚885?D5EÍòÇ trainer/global_step‚10949 % _timestamp‚1.7845063124400125e+09 _runtime‚4466.504612948 _step‚885 & lr-AdamW/pg1‚9.462683326646811e-05 & lr-AdamW/pg2‚9.462683326646811e-05õæd`ü2ó- _wandbruntime‚4466  _runtime‚4466©p)gÝ'ô×O UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚429513851402240 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110960223 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚402908984751104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚429513956463616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7698190311424 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83256021 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110960223 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76307260 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚410833075912192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7845063124406393e+09 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚416348751538928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚10949 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83256021 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28106292 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7924066257408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚429513956463616 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110970366 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 194226389 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83256023 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48200727 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7674050514018 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7674136662492 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚429513851402240 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83256023 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7923881649664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚421815749112832 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚421815766152192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7698102289408 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76307016 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚424022905240780 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808ŸÈQâ'õÜO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚429513956463616 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7698190311424 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83256021 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 110960223 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76307260 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚410833075912192 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 % _timestamp‚1.7845063124406393e+09 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚416348751538928 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚10949 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83256021 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28106292 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7924066257408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚429513956463616 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110970366 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 194226389 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83256023 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48200727 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7674050514018 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7674136662492 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚429513851402240 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83256023 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7923881649664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚421815749112832 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚421815766152192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7698102289408 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76307016 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚424022905240780 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚429513851402240 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110960223 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚402908984751104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚241 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712özýË#2ö- _wandbruntime‚4466  _runtime‚4466Åcmç&÷áM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7698780399104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48205014 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7674812614048 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚421853115480064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194243622 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚416385623606688 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚424060453260096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7924764597248 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83263344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28108994 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7698868428288 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 194233463 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76313772 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194243622 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚429552000947712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28108758 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚402944704812032 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚416385640646048 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110980278 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48205017 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7845063126979413e+09 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚421853115480064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76314011 _runtime‚4466.761313918 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7698868428288 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚887 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110980278 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚402944679908352 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚424060350072150 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚429552000947712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83263342 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7698780399104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚421853132519424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 trainer/global_step‚10949 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 194233463 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83263342 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚410869262002176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7924582093824 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104Šæ¾ øæM BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚887 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110980278 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚402944679908352 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚424060350072150 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚429552000947712 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83263342 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7698780399104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚421853132519424 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 trainer/global_step‚10949 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 194233463 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83263342 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚410869262002176 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7924582093824 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚239 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7698780399104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48205014 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7674812614048 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚421853115480064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194243622 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚416385623606688 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚424060453260096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7924764597248 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83263344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28108994 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7698868428288 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 194233463 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76313772 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194243622 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚429552000947712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28108758 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚402944704812032 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚416385640646048 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 110980278 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚236 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48205017 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7845063126979413e+09 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚421853115480064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76314011 _runtime‚4466.761313918 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7698868428288 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482÷ŸQF˜2ù- _wandbruntime‚4466  _runtime‚4466-ùÂ~ ú‡ 0loss/FBt2/detection_loss‚0.43806657467414173 trainer/global_step‚10949 2loss/SRqqt1/assignment_loss‚0.6931839767367031 4loss/FBt2/assignment_loss‚0.00029784819697395746 /loss/FRt1/detection_loss‚0.5396386488083229 0loss/SRqqt1/detection_loss‚1.123920754403098 % _timestamp‚1.7845063126993606e+09 _step‚888 0loss/FRt2/assignment_loss‚0.8726331682935837 4loss/FBt1/assignment_loss‚0.00036558515870252804 /loss/FBt1/detection_loss‚0.9483730509721985 _runtime‚ 4466.76225033 epoch‚13 %loss/total_loss‚9.315803858703562 0loss/FRt1/assignment_loss‚2.1195908035082747 2loss/SRqqt2/assignment_loss‚0.6220248066459165 .loss/FRt2/detection_loss‚1.400400321209623 1loss/SRqqt2/detection_loss‚0.5573083200960243Š}jûŒ 0loss/FRt1/assignment_loss‚2.1195908035082747 2loss/SRqqt2/assignment_loss‚0.6220248066459165 .loss/FRt2/detection_loss‚1.400400321209623 1loss/SRqqt2/detection_loss‚0.5573083200960243 0loss/FBt2/detection_loss‚0.43806657467414173 trainer/global_step‚10949 2loss/SRqqt1/assignment_loss‚0.6931839767367031 4loss/FBt2/assignment_loss‚0.00029784819697395746 /loss/FRt1/detection_loss‚0.5396386488083229 0loss/SRqqt1/detection_loss‚1.123920754403098 % _timestamp‚1.7845063126993606e+09 _step‚888 0loss/FRt2/assignment_loss‚0.8726331682935837 4loss/FBt1/assignment_loss‚0.00036558515870252804 /loss/FBt1/detection_loss‚0.9483730509721985 _runtime‚ 4466.76225033 epoch‚13 %loss/total_loss‚9.315803858703562øT¹gVoü‚(:ä ËÏõÒÁÚì¿* proc.memory.percent‚17.542600631713867 cpu‚26.427115562389094) disk./.usagePercent‚76.47459138548155% disk./.usageGB‚1174.0761642456055 disk.nvme1n1.in‚ 13.578125# disk.dm-0.out‚44697.37841796875 network.recv‚ 1.26907e+06 disk.dm-0.in‚ 149.203125 disk.rbd0.in‚0 disk.nvme0c0n1.in‚135.625( disk.nvme0c0n1.out‚36346.01123046875 disk.nvme0n1.in‚135.625" proc.memory.rssMB‚ 2874.1796875 proc.cpu.threads‚17% disk.nvme1n1.out‚12885.0927734375 disk.md0.in‚0 disk.dm-1.in‚0! disk.dm-1.out‚4216.1630859375 disk.rbd0.out‚0& disk.nvme0n1.out‚36346.01123046875$ memory_percent‚99.22597408294678( proc.memory.availableMB‚ 126.81640625 network.sent‚ 1.4528197e+07 disk.nvme1c1n1.in‚ 13.578125' disk.nvme1c1n1.out‚12885.0927734375 disk.md0.out‚ 149.6484375Èx,›ý‚(: ËÏõÒû‘æ¿ gpu.0.gpu‚53.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚47.0 gpu.0.powerWatts‚193.964' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚64.65466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êÊéÈoþ‚(:ä ÚÏõÒ¾èÙ¿* proc.memory.percent‚17.542600631713867 proc.cpu.threads‚17% disk.nvme0n1.out‚36355.5537109375 disk.md0.out‚ 149.94140625 disk.dm-0.in‚ 149.203125( proc.memory.availableMB‚ 126.56640625 cpu‚26.425526674151858$ disk./.usageGB‚1174.076187133789 disk.rbd0.out‚0 disk.nvme0n1.in‚135.625 disk.nvme1c1n1.in‚ 13.578125( disk.nvme1c1n1.out‚12885.96826171875& disk.nvme1n1.out‚12885.96826171875! disk.dm-0.out‚44707.076171875 disk.dm-1.in‚0 network.recv‚ 1.273351e+06) disk./.usagePercent‚76.47459287632562 disk.rbd0.in‚0 disk.nvme0c0n1.in‚135.625' disk.nvme0c0n1.out‚36355.5537109375 disk.md0.in‚0" disk.dm-1.out‚4216.25634765625$ memory_percent‚99.22749996185303 network.sent‚ 1.4589435e+07 disk.nvme1n1.in‚ 13.578125" proc.memory.rssMB‚ 2874.1796875*€¤[›ÿ‚(: ÚÏõÒ…Ÿ¢À gpu.0.gpu‚63.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚112.255' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.41833333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Iö2€- _wandbruntime‚4485  _runtime‚4485[žaÈ & lr-AdamW/pg1‚9.446366287902554e-05 & lr-AdamW/pg2‚9.446366287902554e-05 trainer/global_step‚10999 % _timestamp‚1.7845063317942085e+09 _runtime‚4485.857383807 _step‚889Ë9IPÍ‚Ç & lr-AdamW/pg1‚9.446366287902554e-05 & lr-AdamW/pg2‚9.446366287902554e-05 trainer/global_step‚10999 % _timestamp‚1.7845063317942085e+09 _runtime‚4485.857383807 _step‚889ù=uƒ¼2ƒ- _wandbruntime‚4485  _runtime‚4485ÞOµsÜ'„ÖO N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83622073 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7959019919360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111456975 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111456975 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 trainer/global_step‚10999 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28230251 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚412652590508032 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7707879989426 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83622071 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚404693742613504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚423684067474432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚431416212567040 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚423684084513792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚423684067474432 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172024832 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚431416107505664 $ _timestamp‚1.784506331795198e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 195079046 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7732128053248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7707966137900 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48415224 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48415227 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚412652787436544 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83622073 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚423684084513792 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚431416107505664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83622071 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚418192354926928 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196928512 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7732040031232 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76645478 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111467118 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚425900338104188 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111467118 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7958847894528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425900234916354 _runtime‚4485.861603205 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚404693767517184 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚412652787436544 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83622073 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚423684084513792 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚431416107505664 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83622071 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚418192354926928 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚252 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196928512 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7732040031232 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76645478 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111467118 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚425900338104188 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 111467118 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7958847894528 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚425900234916354 _runtime‚4485.861603205 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚404693767517184 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7959019919360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 111456975 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111456975 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚255 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 trainer/global_step‚10999 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28230251 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚412652590508032 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7707879989426 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83622071 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚404693742613504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚423684067474432 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚431416212567040 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 L=DeviceStatsMonitor.on_train_batch_starÑ¥¥¡\t/active_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚423684084513792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚423684067474432 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 172024832 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚431416107505664 $ _timestamp‚1.784506331795198e+09úÑðˆU2†- _wandbruntime‚4486  _runtime‚4486òæªå&‡ßM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7732806170112 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111477030 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83629392 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚412689184927232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195096265 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111466873 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚425937782935558 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7708642089456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83629392 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7959722252800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7732806170112 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111466873 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7959546040832 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚423721433841664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚404729437770752 # _timestamp‚1.78450633205613e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 195096265 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83629394 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76652163 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7732718140928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83629394 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚431454151982592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚423721450881024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚404729462674432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48419517 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195106424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚418229244034048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7732718140928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚431454257051136 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195106424 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚10999 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚412688983811584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48419514 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28232646 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 _runtime‚4486.119487435 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76651923 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0Ò¯·[ê&ˆäM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚404729437770752 # _timestamp‚1.78450633205613e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 195096265 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83629394 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76652163 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7732718140928 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83629394 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚431454151982592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚423721450881024 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚404729462674432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48419517 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 176211968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195106424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚418229244034048 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7732718140928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚431454257051136 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 195106424 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 trainer/global_step‚10999 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚412688983811584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48419514 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28232646 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 _runtime‚4486.119487435 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76651923 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 201115648 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7732806170112 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111477030 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83629392 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚412689184927232 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195096265 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111466873 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚425937782935558 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7708642089456 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83629392 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7959722252800 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7732806170112 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111466873 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7959546040832 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚423721433841664 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0û4\î2‰- _wandbruntime‚4486  _runtime‚4486FÀ_5Š 1loss/SRqqt2/detection_loss‚0.6140297018685081 % _timestamp‚1.7845063320570946e+09 2loss/SRqqt2/assignment_loss‚0.5887467527191473 3loss/FBt2/assignment_loss‚0.0015901805464411261 1loss/SRqqt1/detection_loss‚0.9995374050499364 /loss/FBt1/detection_loss‚1.1123871459410741 .loss/FBt2/detection_loss‚0.687284996867243 epoch‚13 _runtime‚4486.119936395 0loss/FRt1/assignment_loss‚2.0409975150798227 /loss/FRt2/assignment_loss‚0.812805223511532 0loss/SRqqt1/assignment_loss‚0.63375832758921 %loss/total_loss‚9.491513453789826 trainer/global_step‚10999 _step‚892 2loss/FBt1/assignment_loss‚0.020465902721858022 /loss/FRt2/detection_loss‚1.4401087483711308 .loss/FRt1/detection_loss‚0.539801553523923„fè ‹† .loss/FBt2/detection_loss‚0.687284996867243 epoch‚13 _runtime‚4486.119936395 0loss/FRt1/assignment_loss‚2.0409975150798227 /loss/FRt2/assignment_loss‚0.812805223511532 0loss/SRqqt1/assignment_loss‚0.63375832758921 %loss/total_loss‚9.491513453789826 trainer/global_step‚10999 _step‚892 2loss/FBt1/assignment_loss‚0.020465902721858022 /loss/FRt2/detection_loss‚1.4401087483711308 .loss/FRt1/detection_loss‚0.539801553523923 1loss/SRqqt2/detection_loss‚0.6140297018685081 % _timestamp‚1.7845063320570946e+09 2loss/SRqqt2/assignment_loss‚0.5887467527191473 3loss/FBt2/assignment_loss‚0.0015901805464411261 1loss/SRqqt1/detection_loss‚0.9995374050499364 /loss/FBt1/detection_loss‚1.1123871459410741üs“qeŒ‚(:Ú éÏõÒ¼‹‡À disk.rbd0.in‚0 disk.nvme0c0n1.in‚135.625 disk.nvme0n1.in‚135.625( disk.nvme1c1n1.out‚12889.56689453125" proc.memory.rssMB‚ 2874.1796875 proc.cpu.threads‚17 disk.nvme1n1.in‚ 13.578125& disk.nvme1n1.out‚12889.56689453125! disk.dm-0.out‚44719.353515625# disk.nvme0n1.out‚36365.11328125* proc.memory.percent‚17.542600631713867 cpu‚26.423997555106375' disk./.usagePercent‚76.473186016471 disk.md0.in‚0 disk.dm-0.in‚ 149.203125" disk.dm-1.out‚4216.32275390625% disk.nvme0c0n1.out‚36365.11328125 disk.nvme1c1n1.in‚ 13.578125$ memory_percent‚99.22463893890381( proc.memory.availableMB‚ 127.03515625 network.sent‚ 1.4608237e+07$ disk./.usageGB‚1174.054588317871 disk.md0.out‚150.3125 disk.rbd0.out‚0 network.recv‚ 1.276771e+06 disk.dm-1.in‚0Xk¢7š‚(: éÏõÒõôµÀ gpu.0.gpu‚58.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚117.19' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚39.06333333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚06 92Ž- _wandbruntime‚4505  _runtime‚4505E¤GÈ _runtime‚4505.248590979 _step‚893 & lr-AdamW/pg1‚9.429987743516245e-05 & lr-AdamW/pg2‚9.429987743516245e-05 trainer/global_step‚11049 % _timestamp‚1.7845063511839724e+09»ÑpÍÇ & lr-AdamW/pg1‚9.429987743516245e-05 & lr-AdamW/pg2‚9.429987743516245e-05 trainer/global_step‚11049 % _timestamp‚1.7845063511839724e+09 _runtime‚4505.248590979 _step‚893ý®|<2‘- _wandbruntime‚4505  _runtime‚4505iÚ¡”’×O PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83988123 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚433318468670464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48629724 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚433318468670464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚420035958314928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111953727 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚425552385836032 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 trainer/global_step‚11049 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7765977773056 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76983376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚425552402875392 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚414472542542848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7994017163264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚425552385836032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7766065795072 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48629727 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚414472330934272 MGDeviceStatsMonitor.W - Ù on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7766065795072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚433318363609088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28353917 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76983644 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚406478525379584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83988121 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195951993 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚427777667779762vëÏzâ'“ÜO QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111953727 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚425552385836032 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 trainer/global_step‚11049 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7765977773056 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76983376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚425552402875392 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚414472542542848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7994017163264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚425552385836032 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7766065795072 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48629727 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚414472330934272 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7766065795072 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚433318363609088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28353917 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76983644 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚406478525379584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83988121 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 195951993 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚427777667779762 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83988123 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚433318468670464 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48629724 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚433318468670464 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚420035958314928 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0þ2å—Å2”- _wandbruntime‚4505  _runtime‚4505@Äzç&•áM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _runtime‚4505.505322544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76990114 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195959067 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28356342 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚420072847422048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚425589769242624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83995444 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 % _timestamp‚1.7845063514419782e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚406514195633152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7742471564864 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195969226 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚427815318986912 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111973782 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111963625 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111963625 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚433356513154560 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7994528678400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚420072830382688 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48634014 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚425589752203264 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7766743911936 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚433356513154560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7766743911936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7994713278976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚425589752203264 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚406514220536832 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76990359 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚11049 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238¨a\÷ì&–æM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 _step‚895 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 % _timestamp‚1.7845063514419782e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚406514195633152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚242 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7742471564864 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 195969226 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚427815318986912 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 111973782 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 111963625 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111963625 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚433356513154560 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7994528678400 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚420072830382688 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48634014 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚425589752203264 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7766743911936 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚433356513154560 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7766743911936 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7994713278976 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚425589752203264 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚406514220536832 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76990359 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 trainer/global_step‚11049 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 _runtime‚4505.505322544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76990114 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 195959067 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28356342 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚420072847422048 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚425589769242624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83995444 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7800003536896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚421879578742288 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112460622 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 196804650 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112450479 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _runtime‚4524.619184348 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77320765 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201122816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚435220619712512 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚427420721236992 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7775625088716 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7775538940242 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112460622 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84354171 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚427420721236992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7800003536896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7799915514880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28476783 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚429655100643170 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚435220619712512 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚416292041118208 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84354171 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚408263283241984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77321010 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚435220724773888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚408263258338304 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7775538940242 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112460622 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚84354171 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚427420721236992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7800003536896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7799915514880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28476783 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚429655100643170 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚435220619712512 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚416292041118208 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84354171 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚408263283241984 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77321010 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚435220724773888 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚408263258338304 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7800003536896 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚421879578742288 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112460622 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 196804650 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112450479 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 _runtime‚4524.619184348 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚245 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77320765 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 201122816 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚435220619712512 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚427420721236992 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7775625088716 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚429692751850320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚435258769257984 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚408298978399232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8029481610240 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 % _timestamp‚1.7845063708152533e+09 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196832028 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚427458070564864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84361492 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196832028 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112460377 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7800681653760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77327735 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48848517 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚427458087604224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚427458070564864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7800681653760 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚435258769257984 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112470534 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48848514 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚416328642512896 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 trainer/global_step‚11099 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚408298953495552 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77327495 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ;4D(:êkdeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196832028 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚427458070564864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84361492 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196832028 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112460377 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7800681653760 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77327735 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48848517 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚427458087604224 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚427458070564864 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7800681653760 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚435258769257984 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 112470534 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48848514 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚416328642512896 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 trainer/global_step‚11099 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚408298953495552 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77327495 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚429692751850320 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚435258769257984 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚408298978399232 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8029481610240 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 % _timestamp‚1.7845063708152533e+09 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚437122875815936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7833853256704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49058727 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84720223 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚431532533506578 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚423723182130288 _runtime‚4543.654721265 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚431532636694412 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚437122875815936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚429289022559232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 197667452 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚437122980877312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚418111693679104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112947231 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8063864183296 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84720223 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 197677597 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7833941278720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28598776 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84720221 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7809368415650 % _timestamp‚1.7845063895908916e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112947231 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚11149 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚410048041104384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7809454564124 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77657503 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49058724 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112957374 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330ó¯Oj¯ÜO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚431532636694412 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚437122875815936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚429289022559232 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 197667452 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚437122980877312 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚418111693679104 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112947231 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8063864183296 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚84720223 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 197677597 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7833941278720 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28598776 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84720221 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7809368415650 % _timestamp‚1.7845063895908916e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112947231 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚11149 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚410048041104384 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7809454564124 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77657503 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49058724 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112957374 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 LBDƒ†üþÇeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚437122875815936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7833853256704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49058727 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚84720223 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚264 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 211608576 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚267 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚431532533506578 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚423723182130288 _runtime‚4543.654721265†Y¿d2°- _wandbruntime‚4543  _runtime‚4543Ä,Q[ç&±áM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7810044367094 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84727542 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚11149 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚410083736261632 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚431570184713728 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112957129 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 197694830 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚418148293414912 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚423760037158688 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚429326388926464 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77663963 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112957129 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚429326405965824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7834619395584 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77664202 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49063017 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚431570081525782 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _step‚903 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚437160920292864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84727542 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28601185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197694830 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 196921344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 197684671 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84727544 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚423760054198048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚437161025361408 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84727544 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚429326405965824‹€%iì&²æM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7834531366400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7834619395584 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚431570184713728 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112957129 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 197694830 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚418148293414912 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚423760037158688 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚429326388926464 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77663963 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112957129 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚429326405965824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7834619395584 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚77664202 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49063017 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚431570081525782 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _step‚903 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚437160920292864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚84727542 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28601185 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 197694830 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 196921344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 197684671 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84727544 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚423760054198048 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚437161025361408 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚84727544 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚429326405965824 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7810044367094 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84727542 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚11149 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚410083736261632 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49273227 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85086271 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚431157357960192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113443983 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚411832798966784 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85086273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚439025236980736 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77994901 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7843197891058 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 \HDeviceStatsMonitor.on_}xËktrain_batch_start/allocated_bytes.large_pool.freed‚431157340920832 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚439025131919360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚419931409947136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚433410069557820 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚431157340920832 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚433409966369986 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85086273 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚906 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚425566785518288 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚439025131919360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚419931409947136 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚433410069557820 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚431157340920832 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚433409966369986 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85086273 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚906 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚425566785518288 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49273227 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85086271 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚431157357960192 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113443983 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚411832798966784 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85086273 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚439025236980736 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77994901 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7843197891058 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚431157340920832 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113453881 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85093594 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78001283 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8099330765824 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198547473 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚431194707288064 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚425603657586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7868557137408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚431194724327424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7845064093211582e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚431194707288064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8099521657856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7843873842502 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 trainer/global_step‚11199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚431194724327424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198547473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚439063281464832 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚419968015781888 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚433447617577136 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚439063281464832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7868469108224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49277514 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚439063176396288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85093592 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198557632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚425603640546688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85093594 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28724026 _step‚907 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198557632 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85093592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚1524ýRwÂæM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚439063281464832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7868469108224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49277514 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚439063176396288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85093592 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198557632 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚425603640546688 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85093594 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28724026 _step‚907 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198557632 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85093592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113453881 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85093594 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78001283 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8099330765824 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198547473 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚431194707288064 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚425603657586048 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7868557137408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚431194724327424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 % _timestamp‚1.7845064093211582e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚431194707288064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8099521657856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7843873842502 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 trainer/global_step‚11199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚431194724327424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 215795712 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 198547473 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚439063281464832 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚419968015781888 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚433447617577136‹# …2Ã- _wandbruntime‚4563  _runtime‚4563ÌD¸ Ć /loss/FBt2/detection_loss‚0.4130388181874878 %loss/total_loss‚9.036731471660943 2loss/SRqqt2/assignment_loss‚0.5947945259660804 3loss/FBt1/assignment_loss‚0.0005592248159916236 3loss/FBt2/assignment_loss‚0.0015657235983272955 .loss/FRt1/detection_loss‚0.559831503263532 1loss/SRqqt1/detection_loss‚1.0217084407106343 /loss/FBt1/detection_loss‚0.9916144695721174 _step‚908 epoch‚13 0loss/FRt1/assignment_loss‚1.9245696666387488 0loss/FRt2/assignment_loss‚0.9347272633629686 2loss/SRqqt1/assignment_loss‚0.6210754447915098 % _timestamp‚1.7845064093221734e+09 /loss/FRt2/detection_loss‚1.3126626240452477 trainer/global_step‚11199 _runtime‚4563.385022289 1loss/SRqqt2/detection_loss‚0.6605837667082964¥oø­Å‹ /loss/FRt2/detection_loss‚1.3126626240452477 trainer/global_step‚11199 _runtime‚4563.385022289 1loss/SRqqt2/detection_loss‚0.6605837667082964 /loss/FBt2/detection_loss‚0.4130388181874878 %loss/total_loss‚9.036731471660943 2loss/SRqqt2/assignment_loss‚0.5947945259660804 3loss/FBt1/assignment_loss‚0.0005592248159916236 3loss/FBt2/assignment_loss‚0.0015657235983272955 .loss/FRt1/detection_loss‚0.559831503263532 1loss/SRqqt1/detection_loss‚1.0217084407106343 /loss/FBt1/detection_loss‚0.9916144695721174 _step‚908 epoch‚13 0loss/FRt1/assignment_loss‚1.9245696666387488 0loss/FRt2/assignment_loss‚0.9347272633629686 2loss/SRqqt1/assignment_loss‚0.6210754447915098 % _timestamp‚1.7845064093221734e+09Œÿ±‘øsÆ‚(:è ´ÐõÒżɿ' disk.nvme1c1n1.out‚13431.7626953125 disk.md0.in‚0* proc.memory.percent‚17.543601989746094 network.recv‚ 1.297688e+06% disk./.usageGB‚1173.8183517456055# disk.dm-0.out‚44775.84716796875 disk.dm-1.in‚0 disk.rbd0.in‚0$ memory_percent‚99.22688007354736( proc.memory.availableMB‚ 126.66796875 proc.memory.rssMB‚ 2874.34375 network.sent‚ 1.4850122e+07 disk.dm-1.out‚4753.490234375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚36419.97119140625% disk.nvme1n1.out‚13431.7626953125 cpu‚26.420723879458777 disk.dm-0.in‚ 181.49609375 disk.nvme0c0n1.in‚135.625& disk.nvme0n1.out‚36419.97119140625! disk.nvme1c1n1.in‚ 45.87109375 disk.nvme1n1.in‚ 45.87109375 disk.md0.out‚ 151.8671875 proc.cpu.threads‚17) disk./.usagePercent‚76.45779851787042 disk.nvme0n1.in‚135.625ÄMà œÇ‚(:‘ ´ÐõÒÕèŸÀ gpu.0.gpu‚65.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚111.015' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚37.004999999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0G\2È- _wandbruntime‚4584  _runtime‚4584Ci÷;®=ɨ{ ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.736883819103241 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14852245151996613 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7145167589187622 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990079998970032 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5807833671569824 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7"Purity/*FRt*SRqqt2FBt/event_purity‚0.87385094165802 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988510608673096 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5327805876731873 +jet/accuracy_3_of_4‚0.09917768090963364 +particle/sensitivity‚0.6334127187728882 'particle/f_score‚0.6448213458061218 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7824786305427551 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6300215125083923 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6444343328475952 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17234764993190765 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.07425817102193832 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.016839679330587387 particle/accuracy_1_of_1‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6076388955116272 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2466641664505005 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3849659860134125 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13678382337093353 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.033279161900281906 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6708670854568481 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4476320147514343 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14971958100795746 particle/accuracy_2_of_4‚1 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.949999988079071 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7034042477607727 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.09172867238521576 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7512952089309692 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.22018149495124817 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9847937822341919 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12302806228399277 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8580827116966248 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.86396324634552 *jet/accuracy_1_of_3‚0.9108499884605408 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9959900975227356 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6902941465377808 particle/accuracy_2_of_3‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6899775266647339 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02092781476676464 jet/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3826392590999603 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9172742962837219 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8064137697219849 *jet/accuracy_2_of_3‚0.5558295249938965 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9882038831710815 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6451628804206848 ,jet/accuracy_4_of_4‚0.016514042392373085 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9764077067375183 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.06188759207725525 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2757420837879181 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8100000023841858 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7627526521682739 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12854808568954468 particle/accuracy_1_of_2‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9464285969734192 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14873065054416656 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.023840824142098427 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8662835955619812 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 'particle/accuracy‚0.850182831287384 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8623932003974915 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1427256315946579 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6048347353935242 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9894726872444153 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16250000894069672 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 *jet/accuracy_2_of_4‚0.8164464831352234 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 / Purity/2FRt*SRqqt1FBt/FRt_purity‚ 0.51953125 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14783233404159546 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.16318951547145844 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02340284362435341 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02092781476676464 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6940613985061646 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.423334538936615 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.27272728085517883 particle/accuracy_1_of_3‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6100619435310364 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_3_of_4‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7593616843223572 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9984127283096313 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4604133367538452 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.0785488709807396 particle/accuracy_4_of_5‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7584396004676819 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.995635449886322 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6986600160598755 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5978735685348511 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15122365951538086 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6499112248420715 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1621413230895996 +particle/specificity‚0.9094630479812622 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10000000149011612 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1071428582072258 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9378280639648438 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9937241077423096 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4688572585582733 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 7"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.86396324634552 +jet/accuracy_3_of_3‚0.07145579904317856 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.024425430223345757 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_5‚1 )loss/val_total_loss‚8.751478862273208 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7306049466133118 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3858024477958679 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5720050930976868 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7507136464118958 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04368812218308449 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0235915444791317 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +jet/accuracy_1_of_1‚0.30191391706466675 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.996418833732605 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8530701994895935 _step‚909 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.15217356383800507 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8962324857711792 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4650000035762787 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8555250763893127 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44694408774375916 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1709292232990265 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 8"Purity/2FRt0SRqqt1FBt/event_purity‚0.402777761220932 % _timestamp‚1.7845064305118914e+09 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12365172058343887 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21217697858810425 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7651717662811279 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7validation_average_jet_accuracy‚0.34911850094795227 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8100000023841858 *jet/accuracy_1_of_4‚0.9656453728675842 /particle/accuracy_4_of_4‚0.9484416246414185 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8530701994895935 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6762005090713501 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6317178010940552 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47366204857826233 _runtime‚ 4584.57548771 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.28615888953208923 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7583242654800415 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_3_of_3‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 epoch‚13 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7439835071563721 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7544928193092346 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013089386746287346 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8067562580108643 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992743134498596 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6501292586326599 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14852245151996613 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9949238896369934 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6026355028152466 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_4‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6552464962005615 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9946699738502502 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6770634055137634 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03030179813504219 jet/accuracy_5_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.994788408279419 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6501292586326599 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007277445401996374 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1071428582072258 jet/accuracy_3_of_5‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11526640504598618 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.16219331324100494 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5661963820457458 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4545454680919647 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12734262645244598 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13803568482398987 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7483077645301819 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10000000149011612 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 trainer/global_step‚11213 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5326348543167114 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9857529997825623 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20819178223609924 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5¤i\/Ê­{ 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.12365172058343887 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21217697858810425 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.7651717662811279 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 7validation_average_jet_accuracy‚0.34911850094795227 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.8100000023841858 *jet/accuracy_1_of_4‚0.9656453728675842 /particle/accuracy_4_of_4‚0.9484416246414185 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8530701994895935 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6762005090713501 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6317178010940552 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47366204857826233 _runtime‚ 4584.57548771 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.28615888953208923 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.7583242654800415 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 particle/accuracy_3_of_3‚1 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 epoch‚13 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7439835071563721 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 8"Purity/0FRt0SRqqt1FBt/event_purity‚0.999047577381134 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7544928193092346 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.013089386746287346 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.8067562580108643 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9992743134498596 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6501292586326599 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14852245151996613 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9949238896369934 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6026355028152466 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_4‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6552464962005615 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9946699738502502 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.6770634055137634 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.03030179813504219 jet/accuracy_5_of_5‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 6 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.994788408279419 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6501292586326599 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.007277445401996374 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.1071428582072258 jet/accuracy_3_of_5‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11526640504598618 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.16219331324100494 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5661963820457458 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4545454680919647 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12734262645244598 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.13803568482398987 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7483077645301819 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.10000000149011612 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 trainer/global_step‚11213 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5326348543167114 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9857529997825623 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20819178223609924 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 8"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.736883819103241 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14852245151996613 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.7145167589187622 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990079998970032 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5807833671569824 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 7"Purity/*FRt*SRqqt2FBt/event_purity‚0.87385094165802 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9988510608673096 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5327805876731873 +jet/accuracy_3_of_4‚0.09917768090963364 +particle/sensitivity‚0.6334127187728882 'particle/f_score‚0.6448213458061218 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7824786305427551 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6300215125083923 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6444343328475952 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.17234764993190765 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.07425817102193832 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.016839679330587387 particle/accuracy_1_of_1‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6076388955116272 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 jet/accuracy_4_of_5‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2466641664505005 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.3849659860134125 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13678382337093353 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.033279161900281906 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.6708670854568481 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_5‚1 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.4476320147514343 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.14971958100795746 particle/accuracy_2_of_4‚1 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.949999988079071 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7034042477607727 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.09172867238521576 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7512952089309692 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.22018149495124817 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9847937822341919 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12302806228399277 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8580827116966248 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 7"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.86396324634552 *jet/accuracy_1_of_3‚0.9108499884605408 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9959900975227356 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6902941465377808 particle/accuracy_2_of_3‚1 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_5‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6899775266647339 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02092781476676464 jet/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993565082550049 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.3826392590999603 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9172742962837219 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.8064137697219849 *jet/accuracy_2_of_3‚0.5558295249938965 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9882038831710815 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.6451628804206848 ,jet/accuracy_4_of_4‚0.016514042392373085 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9764077067375183 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.06188759207725525 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2757420837879181 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.8100000023841858 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7627526521682739 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.12854808568954468 particle/accuracy_1_of_2‚1 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9464285969734192 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14873065054416656 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.023840824142098427 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8662835955619812 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 'particle/accuracy‚0.850182831287384 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8623932003974915 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1427256315946579 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6048347353935242 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9894726872444153 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16250000894069672 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 *jet/accuracy_2_of_4‚0.8164464831352234 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 / Purity/2FRt*SRqqt1FBt/FRt_purity‚ 0.51953125 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14783233404159546 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.16318951547145844 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02340284362435341 6 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999047577381134 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02092781476676464 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6940613985061646 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 6 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.423334538936615 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.27272728085517883 particle/accuracy_1_of_3‚1 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 jet/accuracy_2_of_5‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6100619435310364 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 particle/accuracy_3_of_4‚1 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7593616843223572 7 qB„5¥Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9984127283096313 9"Purity/0FRt2SRqqt*FBt/event_purity‚0.4604133367538452 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.0785488709807396 particle/accuracy_4_of_5‚1 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7584396004676819 6 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.995635449886322 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6986600160598755 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5978735685348511 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15122365951538086 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6499112248420715 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1621413230895996 +particle/specificity‚0.9094630479812622 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.10000000149011612 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.1071428582072258 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9378280639648438 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9937241077423096 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.4688572585582733 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.09090909361839294 7"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.86396324634552 +jet/accuracy_3_of_3‚0.07145579904317856 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 9 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.024425430223345757 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_5_of_5‚0 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_5‚1 )loss/val_total_loss‚8.751478862273208 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7306049466133118 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3858024477958679 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5720050930976868 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7507136464118958 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04368812218308449 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.0235915444791317 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 +jet/accuracy_1_of_1‚0.30191391706466675 6 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.996418833732605 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8530701994895935 _step‚909 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 +jet/accuracy_2_of_2‚0.15217356383800507 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8962324857711792 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4650000035762787 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8555250763893127 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44694408774375916 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1709292232990265 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 8"Purity/2FRt0SRqqt1FBt/event_purity‚0.402777761220932 % _timestamp‚1.7845064305118914e+09 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192Ò1T+pË‚(:å ÃÐõÒÿì׿! disk.nvme1c1n1.in‚ 45.87109375' disk.nvme1c1n1.out‚13432.5712890625* proc.memory.percent‚17.543601989746094 cpu‚26.426247910446534 network.recv‚ 1.301725e+06) disk./.usagePercent‚76.45807357860144 disk.rbd0.out‚0 disk.nvme0c0n1.in‚135.625& disk.nvme0n1.out‚36451.93994140625 disk.nvme1n1.in‚ 45.87109375% disk.nvme1n1.out‚13432.5712890625 disk.md0.in‚0 disk.dm-1.out‚4753.564453125 disk.nvme0n1.in‚135.625 disk.md0.out‚152.4375 disk.dm-0.in‚ 181.49609375# disk.dm-0.out‚44807.33154296875$ memory_percent‚99.22864437103271( proc.memory.availableMB‚ 126.37890625 proc.cpu.threads‚17% disk./.usageGB‚1173.8225746154785( disk.nvme0c0n1.out‚36451.93994140625 proc.memory.rssMB‚ 2874.34375 network.sent‚ 1.4911355e+07 disk.dm-1.in‚0 disk.rbd0.in‚0è^½Þ›Ì‚(: ÃÐõҙ߳À gpu.0.gpu‚55.0 gpu.0.memory‚6+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚200.714' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.90466666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄKñß2Í- _wandbruntime‚4598  _runtime‚4598âÿÊFÇÎÁ $ _timestamp‚1.784506443947553e+09 _runtime‚4598.012607132 _step‚910 & lr-AdamW/pg1‚9.363863770951864e-05 & lr-AdamW/pg2‚9.363863770951864e-05 trainer/global_step‚11249©²ÐïÌÏÆ $ _timestamp‚1.784506443947553e+09 _runtime‚4598.012607132 _step‚910 & lr-AdamW/pg1‚9.363863770951864e-05 & lr-AdamW/pg2‚9.363863770951864e-05 trainer/global_step‚11249Ž;c92Ð- _wandbruntime‚4598  _runtime‚4598WøÝ'Ñ×O GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚441920576993792 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚436276385897141 trainer/global_step‚11249 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85689791 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78507312 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85689789 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8154300268032 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28925759 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚414542955189248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚428378878374640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199807930 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚441920576993792 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚433998339916800 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚428378861335280 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚433998322877440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚441920471932416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚433998322877440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114107996 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114118139 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 199797785 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7922149054976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85689791 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7922237076992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49581807 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚436276385897141 trainer/global_step‚11249 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85689791 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78507312 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85689789 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8154300268032 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28925759 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚414542955189248 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚428378878374640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199807930 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚441920576993792 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚433998339916800 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚254 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚428378861335280 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚433998322877440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚441920471932416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚433998322877440 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114107996 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 114118139 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 199797785 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7922149054976 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85689791 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7922237076992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49581807 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚434035689244672 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114128051 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78513944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28927850 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚434035706284032 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚434035689244672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚436313830728511 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚428415733403040 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114117894 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49586094 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚414578650346496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 % _timestamp‚1.7845064442044263e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 199825163 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚441958516409344 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199825163 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚441958621477888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78514208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚422733675041792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8154999791616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDevuzJÌ iceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85697112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28928111 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7922915193856 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 _step‚912 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85697112 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚422733465537536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚434035689244672 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114128051 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78513944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28927850 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚434035706284032 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚434035689244672 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 209504256 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚436313830728511 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚428415733403040 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114117894 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49586094 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚261 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚414578650346496 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 % _timestamp‚1.7845064442044263e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 199825163 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚441958516409344 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199825163 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚441958621477888 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78514208 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚422733675041792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8154999791616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85697112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28928111 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0Ó0¬2Ö- _wandbruntime‚4598  _runtime‚4598§Í@ ×… 0loss/FRt1/assignment_loss‚2.1726310305386427 1loss/FBt1/assignment_loss‚0.01781303499623732 %loss/total_loss‚9.106768868028498 % _timestamp‚1.7845064442053387e+09 _runtime‚4598.268219174 2loss/FBt2/assignment_loss‚0.006868097069142131 /loss/FRt2/detection_loss‚1.5196368851620639 0loss/FBt2/detection_loss‚0.49631979927534653 epoch‚14 _step‚913 0loss/FRt2/assignment_loss‚0.8586510103360742 2loss/SRqqt1/assignment_loss‚0.6495996127905062 trainer/global_step‚11249 2loss/SRqqt2/assignment_loss‚0.5531899390754574 /loss/FRt1/detection_loss‚0.5553378341526883 1loss/SRqqt1/detection_loss‚1.0374317071519983 1loss/SRqqt2/detection_loss‚0.6032281121082868 /loss/FBt1/detection_loss‚0.6360618053720548Š÷ØŠ /loss/FRt1/detection_loss‚0.5553378341526883 1loss/SRqqt1/detection_loss‚1.0374317071519983 1loss/SRqqt2/detection_loss‚0.6032281121082868 /loss/FBt1/detection_loss‚0.6360618053720548 0loss/FRt1/assignment_loss‚2.1726310305386427 1loss/FBt1/assignment_loss‚0.01781303499623732 %loss/total_loss‚9.106768868028498 % _timestamp‚1.7845064442053387e+09 _runtime‚4598.268219174 2loss/FBt2/assignment_loss‚0.006868097069142131 /loss/FRt2/detection_loss‚1.5196368851620639 0loss/FBt2/detection_loss‚0.49631979927534653 epoch‚14 _step‚913 0loss/FRt2/assignment_loss‚0.8586510103360742 2loss/SRqqt1/assignment_loss‚0.6495996127905062 trainer/global_step‚11249 2loss/SRqqt2/assignment_loss‚0.5531899390754574‘Šjm‘jÙ‚(:ß ÒÐõÒõžõ¿ disk.nvme0n1.in‚135.625 disk.nvme1n1.in‚ 45.87109375 disk.md0.in‚0 disk.rbd0.in‚0 cpu‚26.42459030180748 proc.cpu.threads‚17& disk.nvme1c1n1.out‚13433.615234375 disk.md0.out‚ 152.84765625 disk.dm-0.in‚ 181.49609375 disk.rbd0.out‚0% proc.memory.availableMB‚ 125.71875 network.recv‚ 1.306121e+06% disk./.usageGB‚1173.8228950500488! disk.nvme1c1n1.in‚ 45.87109375" disk.dm-0.out‚44811.7783203125$ memory_percent‚99.23267364501953 proc.memory.rssMB‚ 2874.34375 network.sent‚ 1.4982248e+07$ disk.nvme0n1.out‚36456.412109375$ disk.nvme1n1.out‚13433.615234375 disk.dm-1.in‚0! disk.dm-1.out‚4753.7470703125 disk.nvme0c0n1.in‚135.625& disk.nvme0c0n1.out‚36456.412109375* proc.memory.percent‚17.543601989746094) disk./.usagePercent‚76.45809445041843eÁí—šÚ‚(: ÒÐõÒâžÀ gpu.0.gpu‚61.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚200.75' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.91666666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†„Iè2Û- _wandbruntime‚4617  _runtime‚4617BʦÀÈÜ & lr-AdamW/pg1‚9.347181652945146e-05 & lr-AdamW/pg2‚9.347181652945146e-05 trainer/global_step‚11299 % _timestamp‚1.7845064632653801e+09 _runtime‚4617.329096518 _step‚914qäÏÍÝÇ & lr-AdamW/pg1‚9.347181652945146e-05 & lr-AdamW/pg2‚9.347181652945146e-05 trainer/global_step‚11299 % _timestamp‚1.7845064632653801e+09 _runtime‚4617.329096518 _step‚914’â72Þ- _wandbruntime‚4617  _runtime‚46176G]0Ü'ßÖO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚435866658278400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚438153715572715 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86055841 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29047822 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114604748 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚416327713051648 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29048065 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8189282217984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86055841 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49796304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8189101804544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7956086796800 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚915 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚435866641239040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7956174818816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7956086796800 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200670732 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78844126 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚424517020173312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86055839 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78844372 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚424516814856192 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚416327737955328 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚11299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚438153818760549 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚430222464723280 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7931250849435 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚435866658278400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443822728035840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7931336997909 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114604748 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443822833097216 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚435866641239040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86055839 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚438153818760549 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚430222464723280 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7931250849435 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚435866658278400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚443822728035840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7931336997909 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 114604748 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚443822833097216 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚435866641239040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86055839 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚435866658278400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚438153715572715 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86055841 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29047822 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114604748 OIDevicâÿFeStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚416327713051648 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29048065 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8189282217984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86055841 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49796304 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8189101804544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7956086796800 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚915 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚246 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚435866641239040 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7956174818816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7956086796800 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 200670732 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78844126 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚424517020173312 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0“9ÓÞþ2á- _wandbruntime‚4617  _runtime‚4617í‘Á2æ&âàM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200687965 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114614646 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86063160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚438191366779865 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7956852935680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8189801980416 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7931926800879 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚416363433112576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114624803 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚416363408208896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200677806 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86063160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚443860772512768 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚916 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784506463523829e+09 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚438191263591919 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚430259353830400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7932012949465 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114624803 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7956852935680 trainer/global_step‚11299 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200677806 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚443860877581312 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 _runtime‚4617.587158615 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78851094 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86063162 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78850845 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114614646 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86063162 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚443860877581312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455Ó3Ïþë&ãåM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 200687965 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 200687965 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114614646 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚249 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86063160 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 180406272 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚438191366779865 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7956852935680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 205309952 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8189801980416 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7931926800879 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚416363433112576 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114624803 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚416363408208896 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 200677806 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86063160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚443860772512768 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚916 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 $ _timestamp‚1.784506463523829e+09 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚438191263591919 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚430259353830400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7932012949465 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114624803 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7956852935680 trainer/global_step‚11299 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 200677806 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚443860877581312 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 _runtime‚4617.587158615 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78851094 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86063162 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78850845 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 114614646 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86063162 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚443860877581312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174121984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7990024538624 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7990112560640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚437734976640000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_sta"‰Ã¾rt/reserved_bytes.all.current‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29171262 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚437734976640000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚445724984139264 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 % _timestamp‚1.7845064826070082e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 201533534 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115111643 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚418112470914048 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚440031251623957 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199025664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8224093603328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86421891 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3X¿˜§á'ðÛO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚437734976640000 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚445724984139264 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 % _timestamp‚1.7845064826070082e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 201533534 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115111643 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚418112470914048 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚440031251623957 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 199025664 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8224093603328 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚248 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86421891 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86421889 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚437734959600640 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50010804 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚426336763543040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚432066085150640 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8224267725312 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79182317 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7990024538624 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚440031148436123 trainer/global_step‚11349 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86421891 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 174121984 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7990024538624 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7990112560640 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚437734976640000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29171262 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816—kgó¢2ñ- _wandbruntime‚4636  _runtime‚4636cÖ\gæ&òàM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚445763133684736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8224976312832 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚437772343007232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8224787517952 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7990702648320 _runtime‚4636.934571094 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29173669 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784506482871212e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚426373167287808 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79188763 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚418148166071296 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚440068696455327 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115121555 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚445763133684736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29173931 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚432102940179040 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7990702648320 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7990790677504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86429212 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79189028 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115111398 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚437772325967872 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7965842424873 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7990790677504 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚437772343007232 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚432102957218400 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86429212 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 201540608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 201550767 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 201540608 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86429210 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚440068799643273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 201550767 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚437772325967872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50015097 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7965756276287 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816zžÚ&óåM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86429212 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 201540608 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 201550767 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 201540608 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86429210 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚440068799643273 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 201550767 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚437772325967872 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50015097 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚262 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7965756276287 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚445763133684736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8224976312832 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚437772343007232 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8224787517952 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7990702648320 _runtime‚4636.934571094 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29173669 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.784506482871212e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚42637316728蟧Å7808 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79188763 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚418148166071296 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚440068696455327 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115121555 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚445763133684736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29173931 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚432102940179040 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7990702648320 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7990790677504 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86429212 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79189028 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115111398 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚437772325967872 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚265 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7965842424873 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7990790677504 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚437772343007232 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚432102957218400˜÷þX}2ô- _wandbruntime‚4636  _runtime‚4636ÉnR¶ õ† 0loss/FRt1/assignment_loss‚2.1313374289462987 /loss/FRt2/detection_loss‚1.5090457333047225 0loss/SRqqt1/detection_loss‚1.202628296743083 1loss/SRqqt2/detection_loss‚0.6243826990359785 epoch‚14 % _timestamp‚1.7845064828722072e+09 _step‚921 0loss/FRt2/assignment_loss‚0.9753614115444097 4loss/FBt1/assignment_loss‚0.00036059012452973607 /loss/FRt1/detection_loss‚0.6424573240821342 /loss/FBt1/detection_loss‚1.1223315688315778 %loss/total_loss‚9.946202043611521 trainer/global_step‚11349 2loss/SRqqt1/assignment_loss‚0.6233770263375527 2loss/FBt2/assignment_loss‚0.005438282016538027 /loss/FBt2/detection_loss‚0.5602231373611305 _runtime‚4636.935124395 2loss/SRqqt2/assignment_loss‚0.5492585452835674)$5ö‹ epoch‚14 % _timestamp‚1.7845064828722072e+09 _step‚921 0loss/FRt2/assignment_loss‚0.9753614115444097 4loss/FBt1/assignment_loss‚0.00036059012452973607 /loss/FRt1/detection_loss‚0.6424573240821342 /loss/FBt1/detection_loss‚1.1223315688315778 %loss/total_loss‚9.946202043611521 trainer/global_step‚11349 2loss/SRqqt1/assignment_loss‚0.6233770263375527 2loss/FBt2/assignment_loss‚0.005438282016538027 /loss/FBt2/detection_loss‚0.5602231373611305 _runtime‚4636.935124395 2loss/SRqqt2/assignment_loss‚0.5492585452835674 0loss/FRt1/assignment_loss‚2.1313374289462987 /loss/FRt2/detection_loss‚1.5090457333047225 0loss/SRqqt1/detection_loss‚1.202628296743083 1loss/SRqqt2/detection_loss‚0.6243826990359785™›­ùßo÷‚(:ä ÿÐõÒ¼è¨À disk.rbd0.out‚0 proc.memory.rssMB‚ 2874.34375) disk./.usagePercent‚76.29973102986395% disk./.usageGB‚1171.3916206359863! disk.nvme1c1n1.in‚ 45.87109375 cpu‚26.419899368170405 proc.cpu.threads‚17 network.sent‚ 1.5110962e+07 network.recv‚ 1.317306e+06 disk.nvme0c0n1.in‚135.625 disk.nvme1n1.in‚ 45.87109375& disk.nvme1n1.out‚13495.38427734375 disk.dm-1.in‚0$ memory_percent‚99.23176765441895 disk.nvme0n1.in‚135.625( disk.nvme1c1n1.out‚13495.38427734375 disk.md0.in‚0 disk.md0.out‚ 154.69921875' proc.memory.availableMB‚ 125.8671875* proc.memory.percent‚17.543601989746094( disk.nvme0c0n1.out‚36661.18115234375& disk.nvme0n1.out‚36661.18115234375 disk.dm-0.in‚ 181.49609375 disk.dm-0.out‚ 45073.9453125 disk.dm-1.out‚ 4754.2421875 disk.rbd0.in‚0„ÏÀ·œø‚(:‘ ÿÐõÒΠäÀ gpu.0.gpu‚90.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚111.293' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.09766666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E´Ó•2ù- _wandbruntime‚4655  _runtime‚4655<{ Èú & lr-AdamW/pg1‚9.313637934092979e-05 & lr-AdamW/pg2‚9.313637934092979e-05 trainer/global_step‚11399 % _timestamp‚1.7845065016510377e+09 _runtime‚4655.715600171 _step‚922Âj¨:ÍûÇ & lr-AdamW/pg1‚9.313637934092979e-05 & lr-AdamW/pg2‚9.313637934092979e-05 trainer/global_step‚11399 % _timestamp‚1.7845065016510377e+09 _runtime‚4655.715600171 _step‚922šÙ-xJ2ü- _wandbruntime‚4655  _runtime‚4655¥/XÝ'ý×O 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202386191 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86787939 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8024050302464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8259213614080 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚419897253680128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚439603295001600 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚923 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚439603295001600 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚4655.715668101 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86787939 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8023962280448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29294461 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚439603277962240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50225304 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 trainer/global_step‚11399 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79519765 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8024050302464 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86787941 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115598252 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50225307 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚447627345304064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚447627240242688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29294703 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7845065016516938e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115598252 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115608395 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚439603277962240 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202396336 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚441908684487365 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚441908581299531 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7998909800251ýtóâ'þÜO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202386191 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86787939 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8024050302464 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8259213614080 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚419897253680128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚439603295001600 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚923 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚439603295001600 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚4655.715668101 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86787939 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8023962280448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29294461 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚439603277962240 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50225304 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚242 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 trainer/global_step‚11399 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79519765 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8024050302464 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86787941 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 115598252 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50225307 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚447627345304064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 209511424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚447627240242688 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29294703 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 184607744 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7845065016516938e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115598252 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115608395 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚439603277962240 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202396336 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚441908684487365 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚441908581299531 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7998909800251›­Zá2ÿ- _wandbruntime‚4655  _runtime‚4655È8Õ•€áM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115608150 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚447665284719616 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8024640390144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115618307 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7999671900281 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79526465 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚428192647883264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚419932923933696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115608150 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86795262 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79526739 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚439640644329472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚447665389788160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86795260 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115618307 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚439640644329472 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚428192863678976 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8024640390144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202403410 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚439640661368832 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202403410 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50229594 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚419932948837376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 825991äñ1¥R4841600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8024728419328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚924 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202413569 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29297142 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚441946232506681 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202413569 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8024728419328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚433946543567040 trainer/global_step‚11399 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86795260 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚441946129318735 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚447665389788160 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚433946560606400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86795262 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712ÆÀeTì&æM REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚447665389788160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86795260 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115618307 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚439640644329472 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚428192863678976 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8024640390144 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 202403410 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚439640661368832 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202403410 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50229594 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚419932948837376 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8259914841600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8024728419328 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚271 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚924 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202413569 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29297142 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚441946232506681 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202413569 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8024728419328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚274 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚433946543567040 trainer/global_step‚11399 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86795260 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚441946129318735 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚447665389788160 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚433946560606400 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86795262 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115608150 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚447665284719616 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8024640390144 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115618307 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7999671900281 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79526465 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚428192647883264 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚419932923933696 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 190892032 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115608150 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚86795262 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79526739 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚439640644329472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0œ[Á>2‚- _wandbruntime‚4655  _runtime‚4655sK“× ƒ† %loss/total_loss‚9.362043151333335 .loss/FRt2/assignment_loss‚0.87487573325634 2loss/SRqqt1/assignment_loss‚0.6300814851404682 1loss/SRqqt1/detection_loss‚1.0296959879158654 0loss/FBt2/detection_loss‚0.48217761682878646 epoch‚14 0loss/FRt1/assignment_loss‚2.1279723332215914 3loss/FBt1/assignment_loss‚0.0009369395289581916 3loss/FBt2/assignment_loss‚0.0038162182362587566 _runtime‚4655.974659543 _step‚925 1loss/SRqqt2/detection_loss‚0.6827323378820438 /loss/FBt1/detection_loss‚0.9024135569110513 trainer/global_step‚11399 % _timestamp‚1.7845065019117754e+09 2loss/SRqqt2/assignment_loss‚0.6396457364921362 /loss/FRt1/detection_loss‚0.5541583374884452 /loss/FRt2/detection_loss‚1.4335368684313905¹`l„‹ .loss/FRt2/assignment_loss‚0.87487573325634 2loss/SRqqt1/assignment_loss‚0.6300814851404682 1loss/SRqqt1/detection_loss‚1.0296959879158654 0loss/FBt2/detection_loss‚0.48217761682878646 epoch‚14 0loss/FRt1/assignment_loss‚2.1279723332215914 3loss/FBt1/assignment_loss‚0.0009369395289581916 3loss/FBt2/assignment_loss‚0.0038162182362587566 _runtime‚4655.974659543 _step‚925 1loss/SRqqt2/detection_loss‚0.6827323378820438 /loss/FBt1/detection_loss‚0.9024135569110513 trainer/global_step‚11399 % _timestamp‚1.7845065019117754e+09 2loss/SRqqt2/assignment_loss‚0.6396457364921362 /loss/FRt1/detection_loss‚0.5541583374884452 /loss/FRt2/detection_loss‚1.4335368684313905 %loss/total_loss‚9.362043151333335òáq…‚(:æ ŽÑõÒ‹©Á¿ disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚135.625( disk.nvme0c0n1.out‚36669.34033203125! disk.nvme1c1n1.in‚ 45.87109375 proc.memory.rssMB‚ 2874.34375 proc.cpu.threads‚17 network.sent‚ 1.5159309e+07 disk.md0.out‚ 155.1640625# disk.dm-0.out‚45081.72119140625' proc.memory.availableMB‚ 125.8359375 network.recv‚ 1.321812e+06( disk./.usagePercent‚76.2997446959346$ disk./.usageGB‚1171.391830444336& disk.nvme0n1.out‚36669.34033203125' disk.nvme1c1n1.out‚13496.1474609375% disk.nvme1n1.out‚13496.1474609375 disk.dm-1.in‚0$ memory_percent‚99.23195838928223 disk.nvme0n1.in‚135.625 disk.nvme1n1.in‚ 45.87109375 disk.md0.in‚0 disk.dm-0.in‚ 181.49609375! disk.dm-1.out‚4754.4111328125* proc.memory.percent‚17.543601989746094 cpu‚26.418776540786514ÚÈ-†‚(:„ ŽÑõ҈ȢÀ gpu.0.gpu‚0.0 gpu.0.memory‚0+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚110.751' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚36.917 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a¡˜72‡- _wandbruntime‚4675  _runtime‚4675”“ÕPň¿ trainer/global_step‚11449 # _timestamp‚1.78450652192226e+09 _runtime‚ 4675.98540085 _step‚926 & lr-AdamW/pg1‚9.296776870518268e-05 & lr-AdamW/pg2‚9.296776870518268e-05ø”å¡Ê‰Ä trainer/global_step‚11449 # _timestamp‚1.78450652192226e+09 _runtime‚ 4675.98540085 _step‚926 & lr-AdamW/pg1‚9.296776870518268e-05 & lr-AdamW/pg2‚9.296776870518268e-05žãUÔI2Š- _wandbruntime‚4675  _runtime‚4675 ûûiÝ'‹×O LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 196928512 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 203248993 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚449529496346112 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8057900022272 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8032825424133 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚449529601407488 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8293947225088 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79857804 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50439804 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚927 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116095004 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚441471596323840 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29417997 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚441471596323840 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 _runtime‚4675.986437863 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116095004 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚443786117350773 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 % _timestamp‚1.7845065219233127e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚449529496346112 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79857804 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50439804 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚249 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚927 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116095004 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚441471596323840 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29417997 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚441471596323840 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 _runtime‚4675.986437863 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116095004 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚443786117350773 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 % _timestamp‚1.7845065219233127e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚449529496346112 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚441508962691072 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87161310 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚430012528115200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169920512 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚441508962691072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8033501375689 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87161310 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚449567645891584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50444097 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116104902 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚449567540823040 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚441508979730432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116104902 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203276371 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87161312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79864561 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79864325 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚430012333291008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29420231 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚421717706699776 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8294821415424 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚435790146955040 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87161312 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚421717681796096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚443823562182143 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚435790163994400 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116115059 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8058578131968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 _step‚928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8058666161152 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8033415227103 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194824192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8294651494912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 203266212 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 203276371 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚441508979730432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8058578131968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845065221936424e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 203266212 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8033415227103 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚236 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 194824192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8294651494912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 203266212 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 203276371 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚441508979730432 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8058578131968 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7845065221936424e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 203266212 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚441508962691072 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87161310 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚430012528115200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 169920512 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚441508962691072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8033501375689 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87161310 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚449567645891584 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50444097 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116104902 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚449567540823040 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚441508979730432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116104902 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 203276371 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87161312 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79864561 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79864325 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚233 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚430012333291008 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29420231 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚421717706699776 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8294821415424 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚435790146955040 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87161312 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚421717681796096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚443823562182143 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚435790163994400 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116115059 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8058578131968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 _step‚928 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8058666161152 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 — íƒ2- _wandbruntime‚4676  _runtime‚4676…—„‘Š 2loss/SRqqt2/assignment_loss‚0.6186296837981289 4loss/FBt1/assignment_loss‚0.00012114037168586833 /loss/FRt1/detection_loss‚0.6057184825928683 _runtime‚4676.257549474 0loss/FRt1/assignment_loss‚1.9969612700988282 /loss/FRt2/detection_loss‚1.5432133684025051 /loss/FBt1/detection_loss‚0.5799596504657529 epoch‚14 % _timestamp‚1.7845065221947362e+09 _step‚929 0loss/FRt2/assignment_loss‚0.7906734081663827 2loss/SRqqt1/assignment_loss‚0.5990558079482146 4loss/FBt2/assignment_loss‚0.00010225709470922261 trainer/global_step‚11449 1loss/SRqqt1/detection_loss‚1.0335623875183806 1loss/SRqqt2/detection_loss‚0.6017323617310605 0loss/FBt2/detection_loss‚0.41388842930632924 %loss/total_loss‚8.783618247494847¼cä·’ 4loss/FBt2/assignment_loss‚0.00010225709470922261 trainer/global_step‚11449 1loss/SRqqt1/detection_loss‚1.0335623875183806 1loss/SRqqt2/detection_loss‚0.6017323617310605 0loss/FBt2/detection_loss‚0.41388842930632924 %loss/total_loss‚8.783618247494847 2loss/SRqqt2/assignment_loss‚0.6186296837981289 4loss/FBt1/assignment_loss‚0.00012114037168586833 /loss/FRt1/detection_loss‚0.6057184825928683 _runtime‚4676.257549474 0loss/FRt1/assignment_loss‚1.9969612700988282 /loss/FRt2/detection_loss‚1.5432133684025051 /loss/FBt1/detection_loss‚0.5799596504657529 epoch‚14 % _timestamp‚1.7845065221947362e+09 _step‚929 0loss/FRt2/assignment_loss‚0.7906734081663827 2loss/SRqqt1/assignment_loss‚0.5990558079482146¡Gb Äb“‚(:× ÑõÒÜ©Ó¿ network.sent‚ 1.5222851e+07 network.recv‚ 1.326174e+06( disk./.usagePercent‚76.3001459814636" disk.dm-0.out‚45123.4521484375 disk.dm-1.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.out‚ 36708.84375 disk.nvme0n1.in‚135.625 disk.nvme0c0n1.in‚135.625 disk.nvme0n1.out‚ 36708.84375! disk.nvme1c1n1.in‚ 45.87109375& disk.nvme1c1n1.out‚13502.298828125 disk.nvme1n1.in‚ 45.87109375 disk.md0.in‚0$ memory_percent‚99.23374652862549 proc.memory.rssMB‚ 2874.34375 disk.dm-0.in‚ 181.49609375 disk.rbd0.out‚0$ disk.nvme1n1.out‚13502.298828125( proc.memory.availableMB‚ 125.54296875* proc.memory.percent‚17.543601989746094 cpu‚26.417596465740374 proc.cpu.threads‚17# disk./.usageGB‚1171.39799118042! disk.dm-1.out‚4757.3681640625 disk.md0.out‚ 155.6171875G½Êš”‚(:’ ÑõÒ—潿 gpu.0.gpu‚88.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚115.688' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚38.562666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0äÉrp•‚(:å ¬ÑõÒ›¤é¿& proc.memory.availableMB‚ 125.546875 network.sent‚ 1.528508e+07) disk./.usagePercent‚76.30013877571726 disk.nvme0c0n1.in‚135.625$ memory_percent‚99.23372268676758 network.recv‚ 1.330618e+06% disk./.usageGB‚1171.3978805541992! disk.nvme1c1n1.in‚ 45.87109375 disk.nvme1n1.in‚ 45.87109375$ disk.nvme1n1.out‚13503.455078125 disk.md0.out‚ 156.06640625 disk.dm-1.in‚0 proc.memory.rssMB‚ 2874.34375 disk.nvme0n1.in‚135.625& disk.nvme0n1.out‚36711.22216796875 disk.md0.in‚0 disk.dm-0.in‚ 181.49609375# disk.dm-0.out‚45125.85791015625* proc.memory.percent‚17.543601989746094 cpu‚26.415998783980207 proc.cpu.threads‚17( disk.nvme0c0n1.out‚36711.22216796875& disk.nvme1c1n1.out‚13503.455078125! disk.dm-1.out‚4757.5283203125 disk.rbd0.in‚0 disk.rbd0.out‚0M©þÕ–‚(:… ¬ÑõÒð×Ô¿ gpu.0.gpu‚53.0 gpu.0.memory‚7+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚203.202' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚67.734 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0R@ËÔ2—- _wandbruntime‚4696  _runtime‚4696¶ ?Ș & lr-AdamW/pg1‚9.279856697279655e-05 & lr-AdamW/pg2‚9.279856697279655e-05 trainer/global_step‚11499 % _timestamp‚1.7845065422495847e+09 _runtime‚4696.312748716 _step‚930ƒáꪂ & lr-AdamW/pg2‚9.279856697279655e-05 trainer/global_step‚11499 % _timestamp‚1.7845065422495847e+09 _runtime‚4696.312748716 _step‚930 & lr-AdamW/pg1‚9.279856697279655e-05¢д‡ª2š- _wandbruntime‚4696  _runtime‚4696ñX¢›ÖO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8091925786112 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚451431857510912 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚451431857510912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.fŒÀwØ:!reed‚80195915 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8329121843200 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80196171 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚445663447026347 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116591756 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚451431752449536 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚443339931724800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29541611 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚431795685931008 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 _runtime‚4696.313727817 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50654304 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚443339914685440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8066654899541 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8328941429760 trainer/global_step‚11499 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚443339914685440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87520041 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚431795891248128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚445663550214181 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204111795 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚437596878275280 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29541864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚443339931724800 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 204121940 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116601899 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8091925786112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0GìÉ(á'œÛO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8091925786112 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚451431857510912 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚451431857510912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80195915 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8329121843200 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 180413440 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80196171 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚445663447026347 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 116591756 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚256 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚451431752449536 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚443339931724800 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29541611 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚431795685931008 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 _runtime‚4696.313727817 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50654304 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚443339914685440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚253 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8066654899541 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8328941429760 trainer/global_step‚11499 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚443339914685440 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87520041 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚431795891248128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 205317120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚445663550214181 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 204111795 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚437596878275280 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29541864 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚443339931724800 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 204121940 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 116601899 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8091925786112 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0£+uí°2- _wandbruntime‚4696  _runtime‚4696pýç&žáM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87527360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚443377281052672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50658597 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80202673 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8329639223808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8092603902976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚443377298092032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚443377281052672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚423502464562176 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚932 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚451469901995008 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚437633767382400 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚443377298092032 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204139173 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116611811 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8329821727232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚423502439658496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚437633750343040 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204129014 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80202923 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚451469796926464 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204139173 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116601654 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50658594 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 % _timestamp‚1.7845065425213478e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87527362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29544326 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚11499 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116611811 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8092603902976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116601654 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8092515873792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8092515873792 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚431832078882304 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8067330851097 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚451469901995008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚445701098233497 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0?Z'Z©ŸæM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204129014 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚87527362 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚451469796926464 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 204139173 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116601654 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50658594 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 % _timestamp‚1.7845065425213478e+09 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚87527362 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29544326 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 trainer/global_step‚11499 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116611811 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8092603902976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116601654 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8092515873792 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8092515873792 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚431832078882304 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8067330851097 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚451469901995008 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚445701098233497 KEDeviceStatsWÙüûCMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87527360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚443377281052672 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50658597 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80202673 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8329639223808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8092603902976 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚443377298092032 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚443377281052672 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚423502464562176 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚932 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚451469901995008 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚437633767382400 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚443377298092032 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204139173 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116611811 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8329821727232 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚423502439658496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚437633750343040 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204129014 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80202923 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8100484374949 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7845065630411568e+09 _step‚935 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50868804 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚453334008552960 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8125863527936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29665202 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117088508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50868807 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29664952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚433615428675072 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8363926311424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8125863527936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80534009 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8125775505920 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚439440498702640 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8100398226475 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 204984742 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87886089 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 trainer/global_step‚11549 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚445208250086400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚445208233047040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚425251527267328 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87886091 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚445208233047040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚439440481663280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152ÊÂÎâ'ªÜO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚445208250086400 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚453334113614336 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚87886091 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8125775505920 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚447540983077589 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87886089 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80533756 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚250 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚425251502363648 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚445208233047040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚425251527267328 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚87886091 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚445208233047040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚439440481663280 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚253 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8100484374949 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7845065630411568e+09 _step‚935 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50868804 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚453334008552960 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8125863527936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29665202 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117088508 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50868807 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29664952 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚433615428675072 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8363926311424 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8125863527936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚80534009 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8125775505920 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚439440498702640 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8100398226475 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 204984742 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87886089 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 trainer/global_step‚11549 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚445208250086400 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87893410 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚445245616453632 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204991816 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204991816 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117098406 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚453372053029888 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚453372158098432 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚447578531096905 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8126453615616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8101074177919 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8101160326505 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 H3Κ Ì:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205001975 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87893410 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚439477370770400 _runtime‚4717.380215729 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8126541644800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚445245599414272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80540739 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50873094 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚433652032358400 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚11549 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29667642 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29667391 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚445245599414272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8364621138944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚439477353731040 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _step‚936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8126453615616 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205001975 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117108563 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚453372158098432 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80540485 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚453372158098432 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚447578531096905 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8126453615616 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8101074177919 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8101160326505 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205001975 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚87893410 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚439477370770400 _runtime‚4717.380215729 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8126541644800 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚445245599414272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80540739 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50873094 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚433652032358400 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 188794880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚251 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 trainer/global_step‚11549 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚254 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29667642 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29667391 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚445245599414272 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8364621138944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚439477353731040 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _step‚936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8126453615616 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205001975 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117108563 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚453372158098432 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80540485 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87893410 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚445245616453632 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¨Àæda2®- _wandbruntime‚4717  _runtime‚4717åêí ¯ƒ .loss/FBt2/detection_loss‚0.653022430387757 0loss/FRt2/assignment_loss‚0.8501342414300653 2loss/SRqqt2/assignment_loss‚0.5409656799125449 /loss/FRt2/detection_loss‚1.4151629175016271 trainer/global_step‚11549 /loss/FRt1/detection_loss‚0.5275131815622139 1loss/SRqqt1/detection_loss‚0.9865614097604217 /loss/FBt1/detection_loss‚0.6796027220704633 epoch‚14 %loss/total_loss‚8.987543732270463 /loss/FRt1/assignment_loss‚2.020797042313688 2loss/SRqqt1/assignment_loss‚0.5711095936130732 2loss/FBt1/assignment_loss‚0.000674504527684467 2loss/FBt2/assignment_loss‚0.016284356041964855 1loss/SRqqt2/detection_loss‚0.7257156531489594 _runtime‚4717.380768161 % _timestamp‚1.7845065633179286e+09 _step‚937§Ë"ú°ˆ 2loss/SRqqt1/assignment_loss‚0.5711095936130732 2loss/FBt1/assignment_loss‚0.000674504527684467 2loss/FBt2/assignment_loss‚0.016284356041964855 1loss/SRqqt2/detection_loss‚0.7257156531489594 _runtime‚4717.380768161 % _timestamp‚1.7845065633179286e+09 _step‚937 .loss/FBt2/detection_loss‚0.653022430387757 0loss/FRt2/assignment_loss‚0.8501342414300653 2loss/SRqqt2/assignment_loss‚0.5409656799125449 /loss/FRt2/detection_loss‚1.4151629175016271 trainer/global_step‚11549 /loss/FRt1/detection_loss‚0.5275131815622139 1loss/SRqqt1/detection_loss‚0.9865614097604217 /loss/FBt1/detection_loss‚0.6796027220704633 epoch‚14 %loss/total_loss‚8.987543732270463 /loss/FRt1/assignment_loss‚2.020797042313688©³WKŒk±‚(:à ÊÑõÒ²öù¿ proc.memory.rssMB‚ 2874.34375( disk.nvme0c0n1.out‚36722.80908203125 disk.nvme0n1.in‚135.625" disk.nvme1n1.out‚ 13505.1015625 disk.md0.out‚ 156.76953125# disk.dm-0.out‚45137.23876953125( proc.memory.availableMB‚ 126.00390625) disk./.usagePercent‚76.30014374519749% disk./.usageGB‚1171.3979568481445& disk.nvme0n1.out‚36722.80908203125 disk.md0.in‚0 disk.dm-1.out‚ 4757.859375* proc.memory.percent‚17.543601989746094 cpu‚26.417847862234616 network.sent‚ 1.5351363e+07 disk.nvme1n1.in‚ 45.87109375 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚99.23093318939209 proc.cpu.threads‚17 network.recv‚ 1.338606e+06 disk.nvme0c0n1.in‚135.625! disk.nvme1c1n1.in‚ 45.87109375$ disk.nvme1c1n1.out‚ 13505.1015625 disk.dm-0.in‚ 181.49609375 disk.dm-1.in‚0ÇÊ𛲂(: ÊÑõÒ­øÛÀ gpu.0.gpu‚48.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚112.987' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.66233333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0À­eQ2³- _wandbruntime‚4737  _runtime‚4737êÎŽÆ´À % _timestamp‚1.7845065836010125e+09 _runtime‚4737.664204131 _step‚938 % lr-AdamW/pg1‚9.24584010679681e-05 % lr-AdamW/pg2‚9.24584010679681e-05 trainer/global_step‚11599y´@,˵Š% lr-AdamW/pg2‚9.24584010679681e-05 trainer/global_step‚11599 % _timestamp‚1.7845065836010125e+09 _runtime‚4737.664204131 _step‚938 % lr-AdamW/pg1‚9.24584010679681e-05ª\4ÎŽ2¶- _wandbruntime‚4737  _runtime‚4737ÐMÝ'·×O UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚449418415940997 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88252141 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117595403 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 trainer/global_step‚11599 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚455236369717760 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8159713247744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29788840 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚4737.665287633 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8159801269760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8399074812416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚455236369717760 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚447076551408640 % _timestamp‚1.7845065836020873e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205837399 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88252139 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51083304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚449418312753163 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29789090 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8134313850357 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88252139 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚427036285129728 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88252141 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117595403 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8159713247744 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚447076568448000 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447076551408640 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435435158819328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205847544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚435435359942144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80872144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚455236264656384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚427036260226048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8398898593280 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚435435158819328 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205847544 OIDeviceStatsMonitor.on_train_batch_staYEóL rt/inactive_split.large_pool.current‚3 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚435435359942144 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚80872144 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚455236264656384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚427036260226048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8398898593280 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚447076551408640 % _timestamp‚1.7845065836020873e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205837399 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88252139 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51083304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚449418312753163 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29789090 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8134313850357 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88252139 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚427036285129728 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88252141 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117595403 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8159713247744 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚447076568448000 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚447076551408640 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80878871 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚441320974158400 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205864777 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _step‚940 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚427071980286976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚449455963960313 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117605315 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚441320957119040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8134903653327 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚455274309133312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8160479386624 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚435471548939776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚447113934815232 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8160391357440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚447113917775872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚455274414201856 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88259460 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88259462 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51087597 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8399780254208 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205854618 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚449455860772367 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51087594 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚455274414201856 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117595158 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8399593556480 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0‡‘xë&»åM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29791543 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117595158 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚435471760541184 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚80878871 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚441320974158400 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205864777 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 _step‚940 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚427071980286976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚449455963960313 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117605315 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚441320957119040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8134903653327 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚455274309133312 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8160479386624 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚435471548939776 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚266 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚447113934815232 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8160391357440 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚447113917775872 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚455274414201856 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88259460 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88259462 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚269 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51087597 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8399780254208 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205854618 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚449455860772367 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51087594 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚455274414201856 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117595158 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8399593556480 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205854618 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚80879140 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118092155 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚448944869770240 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚437255090710528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚457138520759808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118082012 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚428821018088448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8434047718400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29912442 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8193650989568 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚457138625821184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8168057177291 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88618189 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169927680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51297807 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚437254895879168 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81210481 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚428821042992128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51297804 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118092155 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118082012 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _step‚943 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚451295848804405 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88618189 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 206710346 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 206700201 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 % _timestamp‚1.7845066044151747e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81210246 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457138520759808 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88618191 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8433877790720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚4758.478186286 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8193739011584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194831360 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 % _timestamp‚1.7845066044151747e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81210246 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚457138520759808 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88618191 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8433877790720 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚4758.478186286 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8193739011584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 194831360 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118092155 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚448944869770240 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚437255090710528 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚457138520759808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118082012 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚428821018088448 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8434047718400 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29912442 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8193650989568 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚457138625821184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8168057177291 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88618189 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 169927680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51297807 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚437254895879168 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81210481 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚428821042992128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚235 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51297804 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118092155 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118082012 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 _step‚943 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚451295848804405 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88618189 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 206710346 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 206700201 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 _step‚944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118102067 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8194329099264 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51302094 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚443164577546400 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 206717420 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚448982236137472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚428856713245696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118091910 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚448982236137472 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8194417128448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚448982253176832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚457176670305280 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118091910 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 206727579 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8434570960384 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚443164560507040 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 206717420 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚428856738149376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51302097 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚457176565236736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29914827 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81217179 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚437291497904640 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚451333396823721 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8194417128448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8434759755264 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8168819277321 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81216921 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88625510 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29915082 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚457176670305280 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚11649 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88625510 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118102067 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚451333293635775³°™ /ËæM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81216921 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88625510 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29915082 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚457176670305280 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚11649 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88625510 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118102067 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚451333293635775_ã>½ % _timestamp‚1.7845066046817076e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚448982253176832 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 _step‚944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118102067 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8194329099264 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51302094 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚443164577546400 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 206717420 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚448982236137472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚258 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚428856713245696 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118091910 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚448982236137472 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8194417128448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚448982253176832 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚457176670305280 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118091910 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 206727579 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8434570960384 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚443164560507040 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 206717420 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚428856738149376 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51302097 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚457176565236736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29914827 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81217179 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚437291497904640 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚451333396823721 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8194417128448 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8434759755264 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8168819277321 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816°$×rÛ2Ì- _wandbruntime‚4758  _runtime‚4758úv˜¿ Í… 2loss/SRqqt1/assignment_loss‚0.5766518175510014 /loss/FRt2/detection_loss‚1.4215810692044906 1loss/SRqqt2/detection_loss‚0.7795838693245536 %loss/total_loss‚8.805683086925836 trainer/global_step‚11649 2loss/SRqqt2/assignment_loss‚0.5487520702427164 3loss/FBt1/assignment_loss‚0.0016560165911352343 2loss/FBt2/assignment_loss‚0.003215551776635665 /loss/FRt1/detection_loss‚0.5386139144289119 1loss/SRqqt1/detection_loss‚0.8361674620158923 /loss/FBt2/detection_loss‚0.4594130411152602 _step‚945 0loss/FRt2/assignment_loss‚0.9296686115264893 /loss/FBt1/detection_loss‚0.5331655767125388 epoch‚14 % _timestamp‚1.7845066046826706e+09 _runtime‚4758.745484181 /loss/FRt1/assignment_loss‚2.177214086436211¼Xë4Ί /loss/FRt1/assignment_loss‚2.177214086436211 2loss/SRqqt1/assignment_loss‚0.5766518175510014 /loss/FRt2/detection_loss‚1.4215810692044906 1loss/SRqqt2/detection_loss‚0.7795838693245536 %loss/total_loss‚8.805683086925836 trainer/global_step‚11649 2loss/SRqqt2/assignment_loss‚0.5487520702427164 3loss/FBt1/assignment_loss‚0.0016560165911352343 2loss/FBt2/assignment_loss‚0.003215551776635665 /loss/FRt1/detection_loss‚0.5386139144289119 1loss/SRqqt1/detection_loss‚0.8361674620158923 /loss/FBt2/detection_loss‚0.4594130411152602 _step‚945 0loss/FRt2/assignment_loss‚0.9296686115264893 /loss/FBt1/detection_loss‚0.5331655767125388 epoch‚14 % _timestamp‚1.7845066046826706e+09 _runtime‚4758.745484181±k^ÜiÏ‚(:Þ ÷ÑõÒšù‰À proc.cpu.threads‚17 network.recv‚ 1.350022e+06 disk.nvme0c0n1.in‚135.625' proc.memory.availableMB‚ 125.6484375% disk./.usageGB‚1169.2081184387207! disk.nvme1c1n1.in‚ 45.87109375( disk.nvme1c1n1.out‚13506.95947265625& disk.nvme1n1.out‚13506.95947265625 disk.dm-0.in‚ 181.49609375 disk.nvme0n1.in‚135.625! disk.nvme0n1.out‚ 36752.859375) disk./.usagePercent‚76.15750649332166 disk.md0.out‚ 157.6953125# disk.dm-0.out‚45166.85107421875 disk.dm-1.in‚0 disk.dm-1.out‚4758.115234375 disk.rbd0.in‚0$ memory_percent‚99.23310279846191* proc.memory.percent‚17.543601989746094 network.sent‚ 1.5494715e+07 disk.nvme1n1.in‚ 45.87109375 disk.md0.in‚0 disk.rbd0.out‚0# disk.nvme0c0n1.out‚ 36752.859375 proc.memory.rssMB‚ 2874.34375 cpu‚26.42327000149852Á,ÐÀ›Ð‚(: ÷ÑõÒÈâü¿ gpu.0.gpu‚22.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚110.879' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.95966666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô—ù2Ñ- _wandbruntime‚4779  _runtime‚4779ˆ/VwÈÒ % _timestamp‚1.7845066253469806e+09 _runtime‚4779.410132262 _step‚946 & lr-AdamW/pg1‚9.211590342014686e-05 & lr-AdamW/pg2‚9.211590342014686e-05 trainer/global_step‚11699LKƒÍÓÇ _runtime‚4779.410132262 _step‚946 & lr-AdamW/pg1‚9.211590342014686e-05 & lr-AdamW/pg2‚9.211590342014686e-05 trainer/global_step‚11699 % _timestamp‚1.7845066253469806e+09²h¸&2Ô- _wandbruntime‚4779  _runtime‚4779£oYÝ'Õ×O M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118578764 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51512307 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192996352 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88984239 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _runtime‚4779.411152004 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88984241 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8227588731392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚459040881924608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118578764 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚430605800854528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚450813205171200 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚459040776863232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51512304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88984241 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚453173281667813 % _timestamp‚1.7845066253480194e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚459040776863232 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8468841195520 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 trainer/global_step‚11699 _step‚947 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30035662 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8227588731392 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚444971291827280 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81548216 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8227676753408 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚453173178479979 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚439074835046400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8227676753408 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118588907 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚450813188131840 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚459040881924608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚430605775950848 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚444971308866640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚450813188131840 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8201972801173 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88984239yè #â'ÖÜO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚439074835046400 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8227676753408 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 118588907 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚450813188131840 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚459040881924608 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚430605775950848 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚444971308866640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚450813188131840 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8201972801173 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚88984239 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 118578764 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51512307 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 192996352 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88984239 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _runtime‚4779.411152004 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88984241 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8227588731392 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚459040881924608 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 118578764 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚430605800854528 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚450813205171200 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚459040776863232 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51512304 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88984241 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚453173281667813 % _timestamp‚1.7845066253480194e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚459040776863232 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8468841195520 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 trainer/global_step‚11699 _step‚947 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30035662 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8227588731392 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚444971291827280 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81548216 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8227676753408 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚453173178479979³y!l2×- _wandbruntime‚4779  _runtime‚4779ð„¬h¦ØàM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207590381 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8202562604143 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81554915 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.I™_±@#small_pool.current‚86148586 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88991560 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118598819 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118588662 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51516594 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚445008180934400 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚450850571538432 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207590381 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88991562 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚459078926408704 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51516597 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8469537424896 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚450850554499072 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚450850554499072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚430641496011776 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8469726219776 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207580222 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88991562 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30038081 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _step‚948 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81554675 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207580222 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚459078926408704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8228354870272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30038318 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8202648752729 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8228266841088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8228354870272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚439111222231552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚430641471108096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚453210726499183 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88991560 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118598819 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚439111008532992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚459078821340160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚445008163895040 _runtime‚4779.684848075 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 trainer/global_step‚11699eºÕ ë&ÙåM OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81554915 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88991560 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118598819 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118588662 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51516594 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚445008180934400 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚450850571538432 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207590381 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88991562 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚459078926408704 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51516597 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8469537424896 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚450850554499072 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚450850554499072 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚430641496011776 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8469726219776 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207580222 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚88991562 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30038081 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 _step‚948 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81554675 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207580222 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚459078926408704 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8228354870272 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30038318 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8202648752729 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8228266841088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8228354870272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚439111222231552 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚430641471108096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚453210726499183 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88991560 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118598819 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚237 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚439111008532992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚459078821340160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚445008163895040 _runtime‚4779.684848075 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 213698560 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 trainer/global_step‚11699 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207590381 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8202562604143´žm2Ú- _wandbruntime‚4779  _runtime‚4779]ßÀ²Û€ 1loss/SRqqt2/assignment_loss‚0.629173151601859 /loss/FRt1/detection_loss‚0.4855726562875584 %loss/total_loss‚9.670433134720318 2loss/SRqqt1/assignment_loss‚0.5968699788352461 /loss/FBt2/detection_loss‚0.4823921401449855 epoch‚14 trainer/global_step‚11699 $ _timestamp‚1.784506625622559e+09 0loss/FRt2/assignment_loss‚0.9498123489675068 1loss/FBt2/assignment_loss‚0.01022058035505863 1loss/SRqqt2/detection_loss‚0.7278723435707816 _step‚949 0loss/FRt1/assignment_loss‚2.1925840145290487 1loss/FBt1/assignment_loss‚0.09078711454387153 .loss/FRt2/detection_loss‚1.694098522219186 1loss/SRqqt1/detection_loss‚1.0462803469862842 /loss/FBt1/detection_loss‚0.7647699366789311 _runtime‚4779.685438876T'Äê Ü… 2loss/SRqqt1/assignment_loss‚0.5968699788352461 /loss/FBt2/detection_loss‚0.4823921401449855 epoch‚14 trainer/global_step‚11699 $ _timestamp‚1.784506625622559e+09 0loss/FRt2/assignment_loss‚0.9498123489675068 1loss/FBt2/assignment_loss‚0.01022058035505863 1loss/SRqqt2/detection_loss‚0.7278723435707816 _step‚949 0loss/FRt1/assignment_loss‚2.1925840145290487 1loss/FBt1/assignment_loss‚0.09078711454387153 .loss/FRt2/detection_loss‚1.694098522219186 1loss/SRqqt1/detection_loss‚1.0462803469862842 /loss/FBt1/detection_loss‚0.7647699366789311 _runtime‚4779.685438876 1loss/SRqqt2/assignment_loss‚0.629173151601859 /loss/FRt1/detection_loss‚0.4855726562875584 %loss/total_loss‚9.670433134720318µp“%ÉiÝ‚(:Þ †ÒõÒ¯ÖÔ¿ cpu‚26.42518164381428 proc.cpu.threads‚17% disk./.usageGB‚1168.9303245544434 disk.dm-1.in‚0! disk.dm-1.out‚4758.2373046875 disk.rbd0.out‚0% disk.nvme1c1n1.out‚13507.78515625 proc.memory.rssMB‚ 2874.34375) disk./.usagePercent‚76.13941211883689" disk.dm-0.out‚45176.1826171875 disk.rbd0.in‚0% disk.nvme0c0n1.out‚36762.73046875 disk.nvme0n1.in‚135.625# disk.nvme0n1.out‚36762.73046875 network.recv‚ 1.354368e+06 disk.dm-0.in‚ 181.49609375 disk.nvme0c0n1.in‚135.625! disk.nvme1c1n1.in‚ 45.87109375 disk.nvme1n1.in‚ 45.87109375 disk.md0.in‚0$ memory_percent‚99.23179149627686* proc.memory.percent‚17.543601989746094 network.sent‚ 1.5543016e+07# disk.nvme1n1.out‚13507.78515625 disk.md0.out‚ 158.28515625( proc.memory.availableMB‚ 125.86328125| ÛÞ‚(: †ÒõÒí¹±À gpu.0.gpu‚64.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚200.917' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚66.97233333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÝrR{qß‚(:æ •ÒõÒÇ‚ù¿# disk.dm-0.out‚45189.13818359375 disk.rbd0.out‚0( disk.nvme0c0n1.out‚36774.65966796875& disk.nvme0n1.out‚36774.65966796875 disk.rbd0.in‚0 disk.nvme0c0n1.in‚135.625 disk.nvme0n1.in‚135.625! disk.nvme1c1n1.in‚ 45.87109375 disk.nvme1n1.in‚ 45.87109375% disk.nvme1n1.out‚13512.5595703125$ memory_percent‚99.23181533813477* proc.memory.percent‚17.543601989746094 network.sent‚ 1.5606578e+07% disk./.usageGB‚1168.9303016662598& proc.memory.availableMB‚ 125.859375 proc.memory.rssMB‚ 2874.34375 cpu‚26.427436820391474 proc.cpu.threads‚17) disk./.usagePercent‚76.13941062799282 disk.dm-1.in‚0! disk.dm-1.out‚4761.4677734375' disk.nvme1c1n1.out‚13512.5595703125 disk.md0.in‚0 disk.md0.out‚ 158.515625 network.recv‚ 1.358448e+06 disk.dm-0.in‚ 181.49609375‹”¹þà‚(:… •ÒõÒµ÷¶À gpu.0.gpu‚57.0 gpu.0.memory‚8+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚110.778' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚36.926 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€ï¢û2á- _wandbruntime‚4800  _runtime‚4800ñÑ™nÈâ & lr-AdamW/pg1‚9.194378704403308e-05 & lr-AdamW/pg2‚9.194378704403308e-05 trainer/global_step‚11749 % _timestamp‚1.7845066461060057e+09 _runtime‚4800.169128311 _step‚950s$÷+ÍãÇ % _timestamp‚1.7845066461060057e+09 _runtime‚4800.169128311 _step‚950 & lr-AdamW/pg1‚9.194378704403308e-05 & lr-AdamW/pg2‚9.194378704403308e-05 trainer/global_step‚11749¶v $2ä- _wandbruntime‚4800  _runtime‚4800ôòè2#å×O ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8261614495232 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51726807 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8235802276581 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119085659 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89350291 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119075516 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460943032966656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8261614495232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8261526473216 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚432390558716928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 trainer/global_step‚11749 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30159199 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚452681523532800 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30159454 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81886003 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51726804 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 208425805 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89350289 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8503797434368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚455050611343387 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚951 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8235716128107 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚452681506493440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89350291 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚460943032966656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8261614495232 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8261526473216 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚432390558716928 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 trainer/global_step‚11749 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚255 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30159199 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚452681523532800 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30159454 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81886003 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51726804 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 208425805 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89350289 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8503797434368 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚455050611343387 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 _step‚951 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8235716128107 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚452681506493440 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚89350291 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚440930725345792 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚455088159362591 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _step‚952 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 208453183 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89357612 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8262204582912 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8262204582912 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89357610 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚455088262550537 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8236478228137 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚452718872860672 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208453183 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30161678 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚460981182512128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚432426253874176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8236392079551 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440930932752896 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8504496375296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8504678878720 _runtime‚4800.439559912 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚460981182512128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119085414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚452718889900032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚452718889900032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚452718872860672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30161922 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 trainer/global_step‚11749 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 % _timestamp‚1.7845066463762295e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89357610 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8262292612096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81892772 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119095571 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89357612 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚460981077443584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119095571 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8262292612096 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 208443024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51731094 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51731097 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208443024 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚432426228970496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0w¾_ì&éæM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚460981182512128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚247 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119085414 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚452718889900032 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚452718889900032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚452718872860672 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30161922 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 trainer/global_step‚11749 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 % _timestamp‚1.7845066463762295e+09 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89357610 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8262292612096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81892772 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119095571 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚89357612 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚460981077443584 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119095571 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8262292612096 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 208443024 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51731094 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51731097 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208443024 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚432426228970496 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚446851784322400 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119085414 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81893019 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚440930725345792 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚455088159362591 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 _step‚952 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 208453183 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89357612 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8262204582912 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚244 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8262204582912 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89357610 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚455088262550537 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8236478228137 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚452718872860672 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208453183 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30161678 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚460981182512128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚432426253874176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8236392079551 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚440930932752896 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8504496375296 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8504678878720 _runtime‚4800.439559912 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544¸êõÜ2ê- _wandbruntime‚4800  _runtime‚4800_ÊcQ ë† 0loss/FRt2/assignment_loss‚0.7160646767719933 2loss/SRqqt2/assignment_loss‚0.5638931621166665 3loss/FBt2/assignment_loss‚0.0018685506462663708 0loss/SRqqt1/detection_loss‚1.052777573184082 2loss/SRqqt1/assignment_loss‚0.5744010759708048 /loss/FRt2/detection_loss‚1.3493753446913932 1loss/SRqqt2/detection_loss‚0.5559687873661197 /loss/FBt1/detection_loss‚0.7844409591828784 0loss/FBt2/detection_loss‚0.29085040291113456 trainer/global_step‚11749 %loss/total_loss‚8.390581872075677 epoch‚14 0loss/FRt1/assignment_loss‚1.9418557386191404 2loss/FBt1/assignment_loss‚0.019220971177626705 /loss/FRt1/detection_loss‚0.5398646294375701 % _timestamp‚1.7845066463772755e+09 _runtime‚4800.440108593 _step‚953WöÞÔøì‹ 0loss/SRqqt1/detection_loss‚1.052777573184082 2loss/SRqqt1/assignment_loss‚0.5744010759708048 /loss/FRt2/detection_loss‚1.3493753446913932 1loss/SRqqt2/detection_loss‚0.5559687873661197 /loss/FBt1/detection_loss‚0.7844409591828784 0loss/FBt2/detection_loss‚0.29085040291113456 trainer/global_step‚11749 %loss/total_loss‚8.390581872075677 epoch‚14 0loss/FRt1/assignment_loss‚1.9418557386191404 2loss/FBt1/assignment_loss‚0.019220971177626705 /loss/FRl0ÝÝt1/detection_loss‚0.5398646294375701 % _timestamp‚1.7845066463772755e+09 _runtime‚4800.440108593 _step‚953 0loss/FRt2/assignment_loss‚0.7160646767719933 2loss/SRqqt2/assignment_loss‚0.5638931621166665 3loss/FBt2/assignment_loss‚0.0018685506462663708¹H‚Eƒ_í‚(:Ô ¤ÒõÒìÅø¿ disk.nvme1n1.in‚ 45.87109375 disk.nvme1n1.out‚ 13513.46875 disk.md0.out‚ 159.07421875 disk.dm-0.in‚ 181.49609375 disk.nvme0c0n1.in‚135.625& proc.memory.availableMB‚ 125.796875 network.recv‚ 1.361768e+06" disk.nvme1c1n1.out‚ 13513.46875" disk.dm-0.out‚45190.0498046875 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚135.625" disk.nvme0n1.out‚ 36776.0546875 disk.dm-1.in‚0 disk.rbd0.out‚0$ memory_percent‚99.23219680786133 cpu‚26.428594103336586 network.sent‚ 1.5611404e+07! disk.nvme1c1n1.in‚ 45.87109375 disk.md0.in‚0! disk.dm-1.out‚4761.6904296875$ disk.nvme0c0n1.out‚ 36776.0546875 proc.memory.rssMB‚ 2874.34375* proc.memory.percent‚17.543601989746094) disk./.usagePercent‚76.12366756308138$ disk./.usageGB‚1168.688606262207ÜL»#œî‚(:‘ ¤ÒõÒÈ•çÀ gpu.0.gpu‚89.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚141.604' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚47.20133333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰…2ï- _wandbruntime‚4820  _runtime‚4820e¥ 1Èð trainer/global_step‚11799 % _timestamp‚1.7845066662882833e+09 _runtime‚4820.352787232 _step‚954 & lr-AdamW/pg1‚9.177109597242498e-05 & lr-AdamW/pg2‚9.177109597242498e-05i´ÃIÍñÇ & lr-AdamW/pg1‚9.177109597242498e-05 & lr-AdamW/pg2‚9.177109597242498e-05 trainer/global_step‚11799 % _timestamp‚1.7845066662882833e+09 _runtime‚4820.352787232 _step‚954ºv'vÿ2ò- _wandbruntime‚4820  _runtime‚4820âª×ÎÝ'ó×O NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚448658498603280 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 % _timestamp‚1.7845066662903788e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119582411 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51941304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚454549841894400 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚11799 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89716339 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8269545603515 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89716339 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚462845394131456 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚456928044206795 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119572268 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚462845394131456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119582411 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8538927959552 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚454549841894400 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8295464215040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434175291675648 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚456928147394629 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82224196 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8295552237056 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚462845289070080 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30282645 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚442714037124608 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚462845289070080 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8269631751989 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434175316579328 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89716341 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚434175316579328 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 207414272 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 182510592 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89716341 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚11799 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚89716339 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8269545603515 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89716339 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚462845394131456 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚456928044206795 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119572268 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚462845394131456 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 119582411 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8538927959552 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚454549841894400 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8295464215040 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚247 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚434175291675648 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚456928147394629 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82224196 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8295552237056 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚462845289070080 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30282645 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816»æå2õ- _wandbruntime‚4820  _runtime‚48208ÍÙ£ç&öáM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8296142324736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119592323 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚454587191222272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 trainer/global_step‚11799 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209315985 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8270221554959 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚454587208261632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51945594 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82230661 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8270307703545 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 209305826 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚434211011736576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209305826 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51945597 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89723662 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119582166 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚454587208261632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119592323 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89723660 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚454587191222272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚456965695413945 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚448695370671040 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30285067 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚456965592225999 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚462883333547008 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚462883438615552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82230913 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119582166 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8296230353920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8296230353920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712˜èªÇ÷æM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 YGDeviceStatsMonitor!„ðÁ%&.on_train_batch_end/active_bytes.small_pool.allocated‚ 8296230353920 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8296230353920 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8296142324736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 119592323 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚454587191222272 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 trainer/global_step‚11799 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 209315985 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8270221554959 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚454587208261632 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51945594 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82230661 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8270307703545 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 209305826 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚434211011736576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 182503424 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 209305826 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51945597 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89723662 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119582166 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚454587208261632 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119592323 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89723660 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚454587191222272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚456965695413945 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚448695370671040 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30285067 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚456965592225999 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚462883333547008 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚462883438615552 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82230913 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 207407104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119582166¼P›2ø- _wandbruntime‚4820  _runtime‚4820mÓ§Éù‚ .loss/FRt2/detection_loss‚1.573631141260596 0loss/SRqqt1/detection_loss‚1.037372063765216 /loss/FBt1/detection_loss‚0.5712452169507742 /loss/FBt2/detection_loss‚0.5698302844653792 _step‚957 /loss/FRt1/assignment_loss‚2.013996052157645 2loss/FBt1/assignment_loss‚0.004074958725033185 epoch‚14 0loss/FRt2/assignment_loss‚0.8885915253014691 2loss/SRqqt2/assignment_loss‚0.5297895588158835 1loss/SRqqt2/detection_loss‚0.6620690288837068 %loss/total_loss‚8.981391248969938 trainer/global_step‚11799 % _timestamp‚1.7845066665628777e+09 _runtime‚4820.625697374 2loss/SRqqt1/assignment_loss‚0.5824756314253299 3loss/FBt2/assignment_loss‚0.0007972768590019122 .loss/FRt1/detection_loss‚0.547518510359905w¿N  ú‡ 0loss/SRqqt1/detection_loss‚1.037372063765216 /loss/FBt1/detection_loss‚0.5712452169507742 /loss/FBt2/detection_loss‚0.5698302844653792 _step‚957 /loss/FRt1/assignment_loss‚2.013996052157645 2loss/FBt1/assignment_loss‚0.004074958725033185 epoch‚14 0loss/FRt2/assignment_loss‚0.8885915253014691 2loss/SRqqt2/assignment_loss‚0.5297895588158835 1loss/SRqqt2/detection_loss‚0.6620690288837068 %loss/total_loss‚8.981391248969938 trainer/global_step‚11799 % _timestamp‚1.7845066665628777e+09 _runtime‚4820.625697374 2loss/SRqqt1/assignment_loss‚0.5824756314253299 3loss/FBt2/assignment_loss‚0.0007972768590019122 .loss/FRt1/detection_loss‚0.547518510359905 .loss/FRt2/detection_loss‚1.573631141260596½ácb‰rû‚(:ç ³ÒõÒ½äü¿ disk.rbd0.in‚0 disk.nvme0c0n1.in‚135.625( disk.nvme0c0n1.out‚38694.10400390625 disk.nvme0n1.in‚135.625& disk.nvme0n1.out‚38694.10400390625 network.sent‚ 1.5686474e+07 disk.rbd0.out‚0* proc.memory.percent‚17.543601989746094 cpu‚26.42700851245842 proc.cpu.threads‚17! disk.nvme1c1n1.in‚ 45.87109375 disk.nvme1n1.in‚ 45.87109375 disk.md0.out‚ 159.46484375# disk.dm-0.out‚47107.91064453125 disk.dm-1.in‚0$ memory_percent‚99.23391342163086) disk./.usagePercent‚76.30353914256827% disk./.usageGB‚1171.4500846862793' disk.nvme1c1n1.out‚13514.2001953125% disk.nvme1n1.out‚13514.2001953125 disk.md0.in‚0& proc.memory.availableMB‚ 125.515625 proc.memory.rssMB‚ 2874.34375 network.recv‚ 1.365506e+06 disk.dm-0.in‚ 181.49609375! disk.dm-1.out‚4761.7646484375H ›ü‚(: ³ÒõÒ·¯§À gpu.0.gpu‚81.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚111.281' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚37.09366666666667 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ó)œä2ý- _wandbruntime‚4840  _runtime‚4840•ô ÞÇþÁ & lr-AdamW/pg1‚9.159783297132062e-05 & lr-AdamW/pg2‚9.159783297132062e-05 trainer/global_step‚11849 $ _timestamp‚1.784506686834996e+09 _runtime‚4840.898138037 _step‚958C3]%ÌÿÆ & lr-AdamW/pg2‚9.159783297132062e-05 trainer/global_step‚11849 $ _timestamp‚1.784506686834996e+09 _runtime‚4840.898138037 _step‚958 & lr-AdamW/pg1‚9.159783297132062e-05¾Óaã×2€ - _wandbruntime‚4840  _runtime‚4840€|BÂÝ' ×O QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120079163 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚444533994992640 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8329401956864 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120069020 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚464747650234880 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90082391 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82561303 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8303461227397 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90082389 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120079163 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚458805477070203 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8573731748864 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30405764 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚458805580258037 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚456418143216640 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8303375078923 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90082389 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90082391 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30405499 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚456418160256000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚444533781286912 % _timestamp‚1.7845066868359368e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8329401956864 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8573920550912 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚450502101991280 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚456418143216640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚464747545173504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8329489978880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52155807 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚11849 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚464747545173504 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚450502119030640 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82561303 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8303461227397 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90082389 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120079163 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚458805477070203 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8573731748864 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30405764 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚458805580258037 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚456418143216640 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8303375078923 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90082389 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90082391 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚268 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30405499 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚456418160256000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 213705728 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚444533781286912 % _timestamp‚1.7845066868359368e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8329401956864 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8573920550912 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚450502101991280 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚456418143216640 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚464747545173504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8329489978880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52155807 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 trainer/global_step‚11849 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚464747545173504 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚450502119030640 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚456455509583872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210178787 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120078918 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82568244 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚435995744695296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90089710 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120089075 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90089710 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚444570178268672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30407900 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52160097 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚444570381481472 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120089075 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 210168628 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8330080066560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚464785694718976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚458843128277353 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120078918 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82567994 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚458843025089407 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚450538991098400 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚456455526623232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8574433573376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 % _timestamp‚1.7845066871124039e+09 _runtime‚4841.175789654 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8574611882496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚450538974059040 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210178787 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚456455509583872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚456455526623232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30408147 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚435995769598976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8330168095744 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90089712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90089712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52160094 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8304051030367 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 trainer/global_step‚11849 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104`íì&… æM SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120089075 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 210168628 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8330080066560 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚464785694718976 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚458843128277353 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120078918 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82567994 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚458843025089407 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚450538991098400 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚456455526623232 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 203212800 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8574433573376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 % _timestamp‚1.7845066871124039e+09 _runtime‚4841.175789654 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8574611882496 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚450538974059040 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210178787 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚456455509583872 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚456455526623232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30408147 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚435995769598976 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8330168095744 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90089712 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90089712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52160094 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8304051030367 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 trainer/global_step‚11849 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 178309120 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8330080066560 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚456455509583872 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 210178787 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120078918 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82568244 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚435995744695296 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90089710 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120089075 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90089710 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚444570178268672 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30407900 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52160097 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚444570381481472 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946À넱P2† - _wandbruntime‚4841  _runtime‚4841ø5pm ‡ ƒ 2loss/FBt1/assignment_loss‚0.025581021245337374 /loss/FRt1/detection_loss‚0.5454306444292107 /loss/FRt2/detection_loss‚1.4253910277187825 0loss/FBt2/detection_loss‚0.48978550331978316 %loss/total_loss‚9.714531896659777 epoch‚14 _runtime‚4841.176335145 0loss/FRt2/assignment_loss‚0.8482581801414489 2loss/SRqqt1/assignment_loss‚0.6388710808265661 2loss/SRqqt2/assignment_loss‚0.5964281554499791 " _timestamp‚1.7845066871135e+09 _step‚961 0loss/FRt1/assignment_loss‚2.0723119226674878 4loss/FBt2/assignment_loss‚0.00027210808134084076 0loss/SRqqt2/detection_loss‚0.633958567914991 .loss/FBt1/detection_loss‚1.358322188941141 trainer/global_step‚11849 1loss/SRqqt1/detection_loss‚1.0799214959237087¹Éºúˆ ˆ 1loss/SRqqt1/detection_loss‚1.0799214959237087 2loss/FBt1/assignment_loss‚0.025581021245337374 /loss/FRt1/detection_loss‚0.5454306444292107 /loss/FRt2/detection_loss‚1.4253910277187825 0loss/FBt2/detection_loss‚0.48978550331978316 %loss/total_loss‚9.714531896659777 epoch‚14 _runtime‚4841.176335145 0loss/FRt2/assignment_loss‚0.8482581801414489 2loss/SRqqt1/assignment_loss‚0.6388710808265661 2loss/SRqqt2/assignment_loss‚0.5964281554499791 " _timestamp‚1.7845066871135e+09 _step‚961 0loss/FRt1/assignment_loss‚2.0723119226674878 4loss/FBt2/assignment_loss‚0.00027210808134084076 0loss/SRqqt2/detection_loss‚0.633958567914991 .loss/FBt1/detection_loss‚1.358322188941141 trainer/global_step‚11849Áù J±r‰ ‚(:ç ÂÒõÒžÕû¿ proc.memory.rssMB‚ 2874.34375* proc.memory.percent‚17.543601989746094 disk.nvme0c0n1.in‚135.625' disk.nvme1c1n1.out‚13514.9482421875 disk.nvme1n1.in‚ 45.87109375) disk./.usagePercent‚76.30356349302141 disk.dm-1.in‚0( disk.nvme0c0n1.out‚38695.03564453125 disk.md0.in‚0 disk.md0.out‚ 159.8671875 cpu‚26.425638997927575 proc.cpu.threads‚17 network.sent‚ 1.5734917e+07! disk.dm-1.out‚4761.9873046875 disk.rbd0.in‚0 disk.nvme0n1.in‚135.625% disk.nvme1n1.out‚13514.9482421875 disk.dm-0.in‚ 181.49609375$ memory_percent‚99.23415184020996' proc.memory.availableMB‚ 125.4765625 network.recv‚ 1.36946e+06% disk./.usageGB‚1171.4504585266113 disk.rbd0.out‚0& disk.nvme0n1.out‚38695.03564453125! disk.nvme1c1n1.in‚ 45.87109375# disk.dm-0.out‚47108.50439453125òniŠ ‚(:’ ÂÒõÒù–å¿ gpu.0.gpu‚84.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚118.103' gpu.0.enforcedPowerLimitWatts‚300.0) gpu.0.powerPercent‚39.367666666666665 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²þûg‹ ‚(:Ü ÑÒõÒÖÝÓ¿ network.sent‚ 1.579723e+07 network.recv‚ 1.373526e+06) disk./.usagePercent‚76.30055049715475 disk.dm-0.out‚47987.00390625 disk.dm-1.in‚0( proc.memory.availableMB‚ 125.72265625 cpu‚26.42412745086264 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚135.625' disk.nvme0c0n1.out‚39277.1455078125% disk.nvme0n1.out‚39277.1455078125% disk./.usageGB‚1171.4042015075684 disk.dm-0.in‚ 181.49609375 disk.nvme0n1.in‚135.625" disk.nvme1n1.out‚ 13812.0078125 disk.md0.in‚0 proc.memory.rssMB‚ 2874.34375 proc.cpu.threads‚17! disk.dm-1.out‚4762.0849609375! disk.nvme1c1n1.in‚ 45.87109375$ disk.nvme1c1n1.out‚ 13812.0078125 disk.nvme1n1.in‚ 45.87109375 disk.md0.out‚ 160.1328125$ memory_percent‚99.23264980316162* proc.memory.percent‚17.543601989746094â4 ß›Œ ‚(: ÑÒõҧݣÀ gpu.0.gpu‚59.0 gpu.0.memory‚9+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚196.476' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚65.49199999999999 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0~ý=m2 - _wandbruntime‚4861  _runtime‚4861áÈŽ  & lr-AdamW/pg1‚9.142400081587863e-05 & lr-AdamW/pg2‚9.142400081587863e-05 trainer/global_step‚11899 % _timestamp‚1.7845067073382013e+09 _runtime‚4861.402125621 _step‚9622)bÍ Ç & lr-AdamW/pg1‚9.142400081587863e-05 & lr-AdamW/pg2‚9.142400081587863e-05 trainer/global_step‚11899 % _timestamp‚1.7845067073382013e+09 _runtime‚4861.402125621 _step‚962ÂÕ΋2 - _wandbruntime‚4861  _runtime‚4861“ðⵑ ×O QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52370304 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 % _timestamp‚1.7845067073390775e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚458286461578240 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8363339698688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82898554 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 _runtime‚4861.402416592 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120565772 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚452345705379280 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8363339698688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30528250 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8608893444096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52370307 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚460682909933611 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚466649801276928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 211014211 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚446353512042496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120575915 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8608704642048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚452345722418640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8363427720704 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚458286478617600 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90448441 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _step‚963 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90448441 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚458286461578240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8337204554331 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211024356 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90448439 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚466649801276928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚458286478617600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8337290702805 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8363427720704 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚437744832304128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚466649906338304 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120565772 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 trainer/global_step‚11899 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90448439ÁW‡•â'’ ÜO O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 _runtime‚4861.402416592 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120565772 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚452345705379280 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8363339698688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30528250 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8608893444096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52370307 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚460682909933611 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚466649801276928 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 211014211 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚446353512042496 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚10143 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120575915 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8608704642048 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚452345722418640 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8363427720704 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 6159996928 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚458286478617600 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90448441 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 _step‚963 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90448441 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚458286461578240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚10143 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚250 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8337204554331 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211024356 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚90448439 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚466649801276928 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚458286478617600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8337290702805 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8363427720704 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚437744832304128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚466649906338304 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 120565772 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 188802048 trainer/global_step‚11899 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90448439 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚247 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52370304 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 % _timestamp‚1.7845067073390775e+09 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚458286461578240 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8363339698688 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 201411584 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82898554 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0Ãõ¢WÁ2“ - _wandbruntime‚4861  _runtime‚4861o sŒç&” áM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚458323827945472 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90455760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8609401485824 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120585827 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚437780527461376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90455762 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82905215 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8337880505775 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚460720561140761 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚452382577447040 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211041589 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚446390115644928 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90455762 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90455760 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120575670 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚458323844984832 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8364105837568 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8364017808384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7845067076172662e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8364017808384 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚11899 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚460720457952815 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚964 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211041589 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚466687845753856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30530874 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8337966654361 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52374594 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _runtime‚4861.680628426 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120575670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚466687950822400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30530621 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚446389904043520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚458323844984832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120585827 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82905471 4ôº$• æM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211041589 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚466687845753856 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30530874 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8337966654361 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52374594 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _runtime‚4861.680628426 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120575670 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚256 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚466687950822400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30530621 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚446389904043520 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚458323844984832 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120585827 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82905471 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚458323827945472 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90455760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8609401485824 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120585827 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚437780527461376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚253 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90455762 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82905215 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8337880505775 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚460720561140761 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚452382577447040 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211041589 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚446390115644928 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90455762 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90455760 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120575670 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚458323844984832 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8364105837568 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8364017808384 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 % _timestamp‚1.7845067076172662e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8364017808384 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 trainer/global_step‚11899 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚460720457952815 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚964Äi;ü2– - _wandbruntime‚4861  _runtime‚4861÷ — † epoch‚14 2loss/SRqqt1/assignment_loss‚0.6019878869361216 3loss/FBt1/assignment_loss‚0.0025858731504566634 2loss/FBt2/assignment_loss‚0.030543844623207556 /loss/FRt2/detection_loss‚1.4453467344449433 %loss/total_loss‚9.000093674686848 _runtime‚4861.681299818 0loss/FRt2/assignment_loss‚0.9481003489345312 2loss/SRqqt2/assignment_loss‚0.6218954860796501 trainer/global_step‚11899 $ _timestamp‚1.784506707618413e+09 _step‚965 1loss/SRqqt1/detection_loss‚1.1143449638264649 /loss/FBt1/detection_loss‚0.5117396607063711 0loss/FBt2/detection_loss‚0.36095377863173117 0loss/FRt1/assignment_loss‚2.0203773600714547 /loss/FRt1/detection_loss‚0.6068560226753663 1loss/SRqqt2/detection_loss‚0.7353617146065496“¯´Š˜ ‹ $ _timestamp‚1.784506707618413e+09 _step‚965 1loss/SRqqt1/detection_loss‚1.1143449638264649 /loss/FBt1/detection_loss‚0.5117396607063711 0loss/FBt2/detection_loss‚0.36095377863173117 0loss/FRt1/assignment_loss‚2.0203773600714547 /loss/FRt1/detection_loss‚0.6068560226753663 1loss/SRqqt2/detection_loss‚0.7353617146065496 epoch‚14 2loss/SRqqt1/assignment_loss‚0.6019878869361216 3loss/FBt1/assignment_loss‚0.0025858731504566634 2loss/FBt2/assignment_loss‚0.030543844623207556 /loss/FRt2/detection_loss‚1.4453467344449433 %loss/total_loss‚9.000093674686848 _runtime‚4861.681299818 0loss/FRt2/assignment_loss‚0.9481003489345312 2loss/SRqqt2/assignment_loss‚0.6218954860796501 trainer/global_step‚11899Åãâ°k™ ‚(:à àÒõÒÄ–¥À network.recv‚ 1.377072e+06) disk./.usagePercent‚76.30161322050307 disk.md0.out‚ 160.62109375 disk.dm-0.out‚ 47995.7578125 disk.dm-1.in‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚39286.3251953125% disk.nvme0n1.out‚39286.3251953125 network.sent‚ 1.5816273e+07% disk./.usageGB‚1171.4205169677734 disk.dm-0.in‚ 181.49609375 disk.rbd0.out‚0 disk.nvme0n1.in‚135.625& disk.nvme1c1n1.out‚13812.751953125 disk.nvme1n1.in‚ 45.87109375$ disk.nvme1n1.out‚13812.751953125 disk.md0.in‚0! disk.dm-1.out‚4762.2236328125 disk.nvme0c0n1.in‚135.625! disk.nvme1c1n1.in‚ 45.87109375* proc.memory.percent‚17.543601989746094 proc.cpu.threads‚17# memory_percent‚99.2361307144165( proc.memory.availableMB‚ 125.15234375 proc.memory.rssMB‚ 2874.34375 cpu‚26.42256333148992–¾5›š ‚(: àÒõÒ¥è¿ gpu.0.gpu‚41.0 gpu.0.memory‚4+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚204.328' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚68.10933333333334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Õîü-2› - _wandbruntime‚4881  _runtime‚4881:ÆÆŠÆœ À trainer/global_step‚11949 # _timestamp‚1.78450672789462e+09 _runtime‚4881.957805231 _step‚966 & lr-AdamW/pg1‚9.124960229037382e-05 & lr-AdamW/pg2‚9.124960229037382e-05á< 1Ë Å & lr-AdamW/pg2‚9.124960229037382e-05 trainer/global_step‚11949 # _timestamp‚1.78450672789462e+09 _runtime‚4881.957805231 _step‚966 & lr-AdamW/pg1‚9.124960229037382e-05ÆIwWò2ž - _wandbruntime‚4881  _runtime‚4881HiäÅÝ'Ÿ ×O NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8371034029739 _runtime‚4881.958830042 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚468552162441728 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚460154796979200 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚462560445984853 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83236067 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚460154779939840 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8643808542720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 trainer/global_step‚11949 % _timestamp‚1.7845067278956933e+09 _step‚967 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121062524 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121062524 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52584807 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚468552057380352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90814491 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30651263 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121072667 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚468552162441728 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚454189325806640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52584804 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚448173187100672 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚454189308767280 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90814491 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8397365462528 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8397365462528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚233 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚439529565262848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚460154779939840 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚439529590166528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211887158 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚101459u]Ïâ'  ÜO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚462560445984853 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83236067 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚460154779939840 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8643808542720 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 trainer/global_step‚11949 % _timestamp‚1.7845067278956933e+09 _step‚967 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 121062524 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 121062524 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚717 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52584807 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 186704896 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚468552057380352 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90814491 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30651263 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 121072667 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚230 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚468552162441728 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚454189325806640 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 5805303808 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚330 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52584804 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚448173187100672 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚454189308767280 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90814491 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8397365462528 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8397365462528 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚233 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚12303 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚439529565262848 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚460154779939840 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚439529590166528 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚330 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 211887158 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 318767104 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8371034029739 _runtime‚4881.958830042 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚468552162441728 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚460154796979200 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0Ç¥Ë 2¡ - _wandbruntime‚4882  _runtime‚4882fsD¨1 ¢ àM K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121072422 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121082579 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121082579 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8644503233024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8644316535296 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90821810 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83243035 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚462597994004169 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚448209576955392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211904391 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚468590101857280 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚468590206925824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚460192163346432 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8398043579392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211894232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8371709981183 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚460192146307072 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚448209788556800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90821812 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83242785 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚439565285323776 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52589097 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚454226197874400 _step‚968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211904391 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121072422 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.784506728173597e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚460192163346432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121082579 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚250 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121082579 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚247 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8644503233024 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8644316535296 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚90821810 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83243035 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚462597994004169 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚448209576955392 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 211904391 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚468590101857280 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚468590206925824 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚460192163346432 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8398043579392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 211894232 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8371709981183 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚460192146307072 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚448209788556800 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90821812 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83242785 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 211601408 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚439565285323776 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52589097 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 186697728 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚454226197874400 _step‚968 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 211904391 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121072422 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 $ _timestamp‚1.784506728173597e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚460192163346432 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 212749960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52799304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚441314323125248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30773967 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚441314348028928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚470454313483776 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8678719874560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚470454313483776 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83573535 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚4902.179457473 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8431303204352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8404949653621 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚449993067903488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚462023115340800 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83573271 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8678537363968 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚470454313483776 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83573535 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _runtime‚4902.179457473 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚17039360 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8431303204352 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚261 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚86148474 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8404949653621 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚449993067903488 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚238 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚462023115340800 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 313566208 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚482 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83573271 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8678537363968 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 212749960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52799304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚441314323125248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 6937378816 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30773967 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚441314348028928 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚470454313483776 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8678719874560 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 6618611712 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚462060481708032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚470492463029248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚464475323679631 _runtime‚4902.456846644 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 trainer/global_step‚11999 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8679236173824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚456069784223040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30776412 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121569174 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83580006 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚462060464668672 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8431981321216 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8431893292032 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52803594 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8405625605177 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91187862 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91187862 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91187860 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91187860 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚470492357960704 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚464475426867577 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _step‚972 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚462060464668672 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121579331 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212757034 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121569174 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚470492463029248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚450029463960576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚462060481708032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8431981321216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52803597 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚441350043186176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8679420774400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212767193 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚450029254456320 $ _timestamp‚1.784506748393461e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121579331 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212767193 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚456069801262400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8405539456591 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212757034 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚464475323679631 _runtime‚4902.456846644 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 318767104 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 trainer/global_step‚11999 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8679236173824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚24903680 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚456069784223040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30776412 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121569174 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83580006 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚10159 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚462060464668672 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8431981321216 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8431893292032 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 318767104 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52803594 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8405625605177 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91187862 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91187862 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 105068544 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91187860 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91187860 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚470492357960704 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚240 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚464475426867577 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 _step‚972 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚462060464668672 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121579331 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212757034 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121569174 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 6937378816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚470492463029248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚450029463960576 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 5878632448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚462060481708032 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8431981321216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52803597 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚441350043186176 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8679420774400 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212767193 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚450029254456320 $ _timestamp‚1.784506748393461e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121579331 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212767193 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚456069801262400 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8405539456591 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212757034 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚243 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 6618611712 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚462060481708032 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚470492463029248 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576Ìšƒ,$2² - _wandbruntime‚4902  _runtime‚4902—í? ³ † 1loss/SRqqt1/assignment_loss‚0.623175972831561 4loss/FBt1/assignment_loss‚0.00018213722717206338 /loss/FRt2/detection_loss‚1.5134786618960425 1loss/SRqqt2/detection_loss‚0.6904460115083849 0loss/FRt1/assignment_loss‚1.9275427584745446 0loss/FRt2/assignment_loss‚0.9269539699020227 2loss/SRqqt2/assignment_loss‚0.6198479319213911 2loss/FBt2/assignment_loss‚0.008462521348382535 /loss/FBt1/detection_loss‚0.4922593146562576 /loss/FBt2/detection_loss‚0.3079353589258206 trainer/global_step‚11999 _runtime‚4902.457471765 /loss/FRt1/detection_loss‚0.5857871908590341 %loss/total_loss‚8.755800306505964 epoch‚14 % _timestamp‚1.7845067483945327e+09 _step‚973 1loss/SRqqt1/detection_loss‚1.0597284769553506K¨'3´ ‹ epoch‚14 % _timestamp‚1.7845067483945327e+09 _step‚973 1loss/SRqqt1/detection_loss‚1.0597284769553506 1loss/SRqqt1/assignment_loss‚0.623175972831561 4loss/FBt1/assignment_loss‚0.00018213722717206338 /loss/FRt2/detection_loss‚1.5134786618960425 1loss/SRqqt2/detection_loss‚0.6904460115083849 0loss/FRt1/assignment_loss‚1.9275427584745446 0loss/FRt2/assignment_loss‚0.9269539699020227 2loss/SRqqt2/assignment_loss‚0.6198479319213911 2loss/FBt2/assignment_loss‚0.008462521348382535 /loss/FBt1/detection_loss‚0.4922593146562576 /loss/FBt2/detection_loss‚0.3079353589258206 trainer/global_step‚11999 _runtime‚4902.457471765 /loss/FRt1/detection_loss‚0.5857871908590341 %loss/total_loss‚8.755800306505964ÍâP¹bµ ‚(:× þÒõÒ²¯þ¿ proc.cpu.threads‚17 network.recv‚ 1.385279e+06 disk.md0.in‚0 disk.md0.out‚ 162.46875 disk.dm-0.out‚ 48030.8359375' proc.memory.availableMB‚ 124.8203125 cpu‚26.419481407367655 disk.dm-1.in‚0! disk.dm-1.out‚4762.4091796875 disk.nvme0c0n1.in‚135.625 disk.nvme0n1.in‚135.625! disk.nvme1c1n1.in‚ 45.87109375 proc.memory.rssMB‚ 2874.34375 network.sent‚ 1.5927953e+07) disk./.usagePercent‚76.30359132211073 disk.dm-0.in‚ 181.49609375' disk.nvme0c0n1.out‚39322.6689453125% disk.nvme0n1.out‚39322.6689453125 disk.nvme1n1.in‚ 45.87109375! disk.nvme1n1.out‚ 13815.515625* proc.memory.percent‚17.543601989746094$ disk./.usageGB‚1171.450885772705 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.nvme1c1n1.out‚ 13815.515625$ memory_percent‚99.23815727233887¸Ø³œ¶ ‚(:‘ þÒõÒ¸‘¯À gpu.0.gpu‚88.0 gpu.0.memory‚11+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚46.0 gpu.0.powerWatts‚192.941' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚64.31366666666666 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µS¥r· ‚(:ç ÓõÒ×Ó¤À% disk.nvme1n1.out‚13816.7373046875 disk.rbd0.in‚0 disk.nvme0n1.in‚135.625 cpu‚26.425144320720392 network.recv‚ 1.388871e+06% disk./.usageGB‚1171.4519309997559 disk.dm-0.in‚ 181.49609375" disk.dm-0.out‚48036.1611328125 disk.rbd0.out‚0! disk.nvme1c1n1.in‚ 45.87109375& proc.memory.availableMB‚ 124.921875 proc.cpu.threads‚17 disk.md0.out‚ 162.92578125! disk.dm-1.out‚4762.5107421875' disk.nvme1c1n1.out‚13816.7373046875# proc.memory.rssMB‚ 2874.67578125* proc.memory.percent‚17.545628547668457 disk.md0.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚135.625' disk.nvme0c0n1.out‚39327.8564453125% disk.nvme0n1.out‚39327.8564453125# memory_percent‚99.2375373840332 network.sent‚ 1.5990326e+07) disk./.usagePercent‚76.30365940398997 disk.nvme1n1.in‚ 45.87109375±d<›¸ ‚(: ÓõÒðÎÕÀ gpu.0.gpu‚48.0 gpu.0.memory‚3+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚110.926' gpu.0.enforcedPowerLimitWatts‚300.0( gpu.0.powerPercent‚36.97533333333333 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ïÂàt2¹ - _wandbruntime‚4924  _runtime‚4924îóÂQº º{ ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9821428656578064 _step‚974 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1434718817472458 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12824338674545288 particle/accuracy_1_of_3‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988389015197754 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1547962725162506 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.007290350738912821 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_3‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9902447462081909 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1628822684288025 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5328499674797058 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9967328310012817 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2770751416683197 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8524171113967896 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9951880574226379 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.06364454329013824 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.16466660797595978 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15599989891052246 particle/accuracy_2_of_4‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6076388955116272 particle/accuracy_5_of_5‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.22066102921962738 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4318181872367859 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7019422054290771 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5930870771408081 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0005668934318237007 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20732879638671875 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 +particle/sensitivity‚0.6219144463539124 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8558897972106934 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5539330244064331 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2481037825345993 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.07648278027772903 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9962593913078308 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5265437960624695 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02092781476676464 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7601292729377747 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0357142873108387 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7466624975204468 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7302411198616028 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9984127283096313 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935 ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0220708716660738 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.05000000074505806 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983673095703125 particle/accuracy_1_of_1‚1 jet/accuracy_4_of_5‚0 % _timestamp‚1.7845067708569882e+09 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9968867301940918 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45630717277526855 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12419503182172775 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912552833557129 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7402203679084778 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0250232070684433 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.717880368232727 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8"Purity/2FRt0SRqqt1FBt/event_purity‚0.402777761220932 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04513212665915489 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 (particle/accuracy‚0.849593818Û?ߪ'1877136 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8486841917037964 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8790578246116638 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1630386859178543 particle/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7406689524650574 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9909163117408752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13779646158218384 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6349033713340759 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.672630786895752 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 +jet/accuracy_1_of_1‚0.30252695083618164 *jet/accuracy_2_of_3‚0.5573165416717529 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.05000000074505806 particle/accuracy_4_of_5‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8533883690834045 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 *jet/accuracy_1_of_3‚0.9061651825904846 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6228210926055908 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6587741374969482 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.45768657326698303 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46353164315223694 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1495683491230011 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 particle/accuracy_2_of_5‚1 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.387247771024704 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7584348917007446 _runtime‚4924.920653573 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_2_of_2‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990079998970032 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6010033488273621 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.385162353515625 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 *jet/accuracy_1_of_4‚0.9678099751472473 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 jet/accuracy_1_of_5‚1 trainer/global_step‚12014 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.0943758636713028 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6876904964447021 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02092781476676464 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5694414377212524 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9905115962028503 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 *jet/accuracy_2_of_4‚0.7994213104248047 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 )loss/val_total_loss‚8.677147240137796 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9967517852783203 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6920559406280518 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.022154321894049644 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3858024477958679 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_4‚1 7validation_average_jet_accuracy‚0.34899449348449707 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6969704627990723 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07293345034122467 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6036744713783264 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9967120289802551 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6537235975265503 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1350572556257248 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6786694526672363 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1495683491230011 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 particle/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983334541320801 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03461526334285736 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2873433530330658 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6537235975265503 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0357142873108387 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8927069902420044 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7782050967216492 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.745222806930542 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44243109226226807 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6526576280593872 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6986180543899536 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.08041450381278992 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7426553964614868 +particle/specificity‚0.9118607640266418 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15041109919548035 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17786121368408203 jet/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1397079974412918 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.679377555847168 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_5‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11636191606521606 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.766272246837616 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9750000238418579 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6052855253219604 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.647958517074585 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.18176676332950592 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.022922322154045105 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21113187074661255 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013902382925152779 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9213560223579407 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5626473426818848 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7481389045715332 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0005668934318237007 . Purity/2FRt*SRqqt1FBt/FRt_purity‚ 0.5390625 *jet/accuracy_2_of_2‚0.1520191878080368 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47052252292633057 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9804894328117371 /particle/accuracy_4_of_4‚0.9315473437309265 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8581196665763855 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6475479006767273 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988840222358704 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7364218831062317 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 ,jet/accuracy_4_of_4‚0.014682539738714695 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9960408806800842 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7923726439476013 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8524171113967896 /particle/accuracy_3_of_4‚0.9965986609458923 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1253889948129654 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983673095703125 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16250000894069672 epoch‚14 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7930340766906738 +jet/accuracy_3_of_4‚0.09871385246515274 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9428544640541077 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8640079498291016 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15929527580738068 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4950000047683716 'particle/f_score‚0.6397153735160828 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8486841917037964 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02302522398531437 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7448218464851379 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4225512146949768 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004178362432867289 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516‘_û,Å=» ¿{ ,&Purity/*FRt2SRqqt2FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9980158805847168 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.0220708716660738 :"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.05000000074505806 *"Purity/2FRt*SRqqt2FBt/event_purity‚0.5 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9983673095703125 particle/accuracy_1_of_1‚1 jet/accuracy_4_of_5‚0 % _timestamp‚1.7845067708569882e+09 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9968867301940918 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.5129278302192688 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.45630717277526855 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.6297432780265808 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.12419503182172775 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_2‚1 <&Purity/*FRt*SRqqt0FBt/event_proportion‚0.881045401096344 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912552833557129 & Purity/*FRt2SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.7402203679084778 9"Purity/2FRt0SRqqt*FBt/event_purity‚0.0250232070684433 ?&Purity/2FRt2SRqqt*FBt/event_proportion‚0.006417410913854837 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.0436662957072258 & Purity/2FRt0SRqqt*FBt/FBt_purity‚1 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.717880368232727 & Purity/2FRt*SRqqt2FBt/FBt_purity‚1 ?&Purity/*FRt*SRqqt2FBt/event_proportion‚0.010997953824698925 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1467633992433548 ("Purity/0FRt1SRqqt2FBt/event_purity‚0 8"Purity/2FRt0SRqqt1FBt/event_purity‚0.402777761220932 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.04513212665915489 <&Purity/2FRt*SRqqt*FBt/event_proportion‚0.040062315762043 (particle/accuracy‚0.8495938181877136 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.8486841917037964 ("Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.8790578246116638 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.4607747495174408 @&Purity/*FRt1SRqqt2FBt/event_proportion‚0.0006975446594879031 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1630386859178543 particle/accuracy_1_of_5‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9537528157234192 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.7406689524650574 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9909163117408752 >&Purity/1FRt0SRqqt1FBt/event_proportion‚0.00969587080180645 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.800000011920929 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.13779646158218384 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.6349033713340759 & Purity/2FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.2839471697807312 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.672630786895752 & Purity/1FRt2SRqqt1FBt/FRt_purity‚0 +jet/accuracy_1_of_1‚0.30252695083618164 *jet/accuracy_2_of_3‚0.5573165416717529 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9444444179534912 :"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.05000000074505806 particle/accuracy_4_of_5‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.8533883690834045 ("Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0 *jet/accuracy_1_of_3‚0.9061651825904846 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.6228210926055908 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.6587741374969482 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.45768657326698303 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.46353164315223694 & Purity/1FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1495683491230011 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.7857142686843872 particle/accuracy_2_of_5‚1 8"Purity/*FRt1SRqqt*FBt/event_purity‚0.387247771024704 & Purity/2FRt*SRqqt*FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7584348917007446 _runtime‚4924.920653573 =&Purity/1FRt1SRqqt*FBt/event_proportion‚0.1534598171710968 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0382486991584301 particle/accuracy_2_of_2‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9990079998970032 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.6010033488273621 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.3181818127632141 ( Purity/2FRt1SRqqt2FBt/FRt_purity‚0.5 8"Purity/*FRt1SRqqt0FBt/event_purity‚0.385162353515625 ?&Purity/0FRt*SRqqt2FBt/event_proportion‚0.008068266324698925 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0462472103536129 *jet/accuracy_1_of_4‚0.9678099751472473 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.8888888955116272 jet/accuracy_1_of_5‚1 trainer/global_step‚12014 7 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.0943758636713028 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.6876904964447021 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.02092781476676464 ("Purity/*FRt1SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5694414377212524 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9905115962028503 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2322126179933548 *jet/accuracy_2_of_4‚0.7994213104248047 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_2_of_5‚1 )loss/val_total_loss‚8.677147240137796 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9967517852783203 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0103004090487957 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.6920559406280518 9 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.022154321894049644 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3670479953289032 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.3858024477958679 @&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0027204242069274187 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.01990327425301075 particle/accuracy_1_of_4‚1 7validation_average_jet_accuracy‚0.34899449348449707 & Purity/2FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.6969704627990723 & Purity/1FRt1SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0127883180975914 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 +jet/accuracy_3_of_3‚0.07293345034122467 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.6036744713783264 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9967120289802551 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.6537235975265503 >&Purity/1FRt*SRqqt1FBt/event_proportion‚0.01643880270421505 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1350572556257248 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.6786694526672363 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2203543484210968 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1495683491230011 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.4408714771270752 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.8500000238418579 particle/accuracy_3_of_5‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.0418526791036129 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9983334541320801 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.03461526334285736 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2873433530330658 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.6537235975265503 & Purity/0FRt2SRqqt*FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt2FBt/event_proportion‚0.0024879092816263437 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9935897588729858 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.0357142873108387 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.8927069902420044 >&Purity/0FRt2SRqqt*FBt/event_proportion‚0.02457682229578495 ,&Purity/1FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.7782050967216492 8"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.745222806930542 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.006417410913854837 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.44243109226226807 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.6526576280593872 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.6986180543899536 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.08041450381278992 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.7426553964614868 +particle/specificity‚0.9118607640266418 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15041109919548035 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.0390159972012043 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.2083565890789032 >&Purity/0FRt2SRqqt0FBt/event_proportion‚0.02385602705180645 & Purity/1FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17786121368408203 jet/accuracy_5_of_5‚0 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.1397079974412918 8"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.679377555847168 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 jet/accuracy_3_of_5‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.11636191606521606 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.766272246837616 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9750000238418579 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.6052855253219604 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.647958517074585 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.18176676332950592 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.0188569575548172 9 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.022922322154045105 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.21113187074661255 ?&Purity/0FRt1SRqqt1FBt/event_proportion‚0.011555989272892475 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013902382925152779 @&Purity/2FRt*SRqqt1FBt/event_proportion‚0.0016043527284637094 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.0617094486951828 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.9213560223579407 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.5626473426818848 @&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0004417782765813172 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.7481389045715332 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0005668934318237007 . Purity/2FRt*SRqqt1FBt/FRt_purity‚ 0.5390625 *jet/accuracy_2_of_2‚0.1520191878080368 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.47052252292633057 & Purity/*FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.9804894328117371 /particle/accuracy_4_of_4‚0.9315473437309265 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.8581196665763855 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.1111111119389534 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.6475479006767273 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9988840222358704 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.7364218831062317 ("Purity/*FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9985994696617126 ,jet/accuracy_4_of_4‚0.014682539738714695 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9960408806800842 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.7923726439476013 @&Purity/2FRt1SRqqt2FBt/event_proportion‚2.325148852833081e-05 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.8524171113967896 /particle/accuracy_3_of_4‚0.9965986609458923 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1253889948129654 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9983673095703125 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9981576204299927 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.16250000894069672 epoch‚14 ("Purity/0FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.6105840802192688 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.7930340766906738 +jet/accuracy_3_of_4‚0.09871385246515274 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.9428544640541077 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.3866024911403656 @&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0009998140158131719 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.8640079498291016 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15929527580738068 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.4950000047683716 'particle/f_score‚0.6397153735160828 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.8486841917037964 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.02302522398531437 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.7448218464851379 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.4225512146949768 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.004178362432867289 @&Purity/2FRt1SRqqt1FBt/event_proportion‚0.0008370535797439516 ,&Purity/2FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9821428656578064 _step‚974 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1434718817472458 @&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0007672990905120969 ,&Purity/2FRt2SRqqt1FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.12824338674545288 particle/accuracy_1_of_3‚1 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9988389015197754 7 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.1547962725162506 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.007290350738912821 A&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00018601190822664648 particle/accuracy_3_of_3‚1 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9934210777282715 ,&Purity/0FRt2SRqqt2FBt/event_proportion‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9902447462081909 @&Purity/0FRt2SRqqt1FBt/event_proportion‚0.0007207961170934141 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1628822684288025 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.5328499674797058 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9967328310012817 <&Purity/*FRt2SRqqt*FBt/event_proportion‚0.054222472012043 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.2770751416683197 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.8524171113967896 & Purity/*FRt2SRqqt1FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.9951880574226379 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.06364454329013824 8&Purity/1FRt2SRqqt0FBt/event_proportion‚ 0.02294921875 @&Purity/2FRt*SRqqt2FBt/event_proportion‚0.0002092633949359879 ?&Purity/2FRt1SRqqt0FBt/event_proportion‚0.011928013525903225 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.16466660797595978 & Purity/2FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.15599989891052246 particle/accuracy_2_of_4‚1 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.6076388955116272 particle/accuracy_5_of_5‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.22066102921962738 ?&Purity/0FRt0SRqqt2FBt/event_proportion‚0.007626487873494625 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4318181872367859 & Purity/1FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.7019422054290771 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.5930870771408081 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0005668934318237007 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.800000011920929 9"Purity/2FRt0SRqqt2FBt/event_purity‚0.5714285969734192 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.20732879638671875 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_3‚1 +particle/sensitivity‚0.6219144463539124 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.8558897972106934 8&Purity/*FRt2SRqqt0FBt/event_proportion‚ 0.05322265625 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5539330244064331 & Purity/0FRt2SRqqt1FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.2481037825345993 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.07648278027772903 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9962593913078308 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0313895083963871 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.5265437960624695 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.02092781476676464 ) Purity/2FRt*SRqqt2FBt/FRt_purity‚0.75 @&Purity/1FRt1SRqqt2FBt/event_proportion‚0.0002325148816453293 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.7601292729377747 A&Purity/1FRt2SRqqt1FBt/event_proportion‚0.00027901786961592734 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.0357142873108387 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0208565853536129 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.7466624975204468 & Purity/2FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.7302411198616028 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9984127283096313 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.4530552327632904 @&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0064639137126505375 >&Purity/1FRt2SRqqt*FBt/event_proportion‚0.02322823740541935Φr¼ ‚(:‚ œÓõÒ›ŸÊ¿ network.sent‚ 1.5993248e+07% disk./.usageGB‚1171.4519958496094 disk.dm-0.out‚48046.54296875 disk.rbd0.out‚0# proc.memory.rssMB‚ 2874.67578125* proc.memory.percent‚17.545628547668457 cpu‚26.426844036854916 proc.cpu.threads‚17 network.recv‚ 1.392537e+06) disk./.usagePercent‚76.30366362804816 disk.dm-0.in‚ 181.49609375$ disk.nvme0c0n1.in‚137.1064453125% disk.nvme0n1.out‚39339.1435546875! disk.nvme1c1n1.in‚ 45.87109375( disk.nvme1c1n1.out‚13817.99658203125 disk.nvme1n1.in‚ 45.87109375' disk.nvme0c0n1.out‚39339.1435546875 disk.md0.in‚ 0.48046875 disk.md0.out‚ 163.71484375 disk.dm-1.in‚0" disk.dm-1.out‚4762.63818359375" disk.nvme0n1.in‚137.1064453125& disk.nvme1n1.out‚13817.99658203125$ memory_percent‚99.23770427703857 disk.rbd0.in‚0( proc.memory.availableMB‚ 124.89453125vy‹k½ ‚(:… œÓõÒ±âªÀ gpu.0.gpu‚51.0 gpu.0.memory‚5+ gpu.0.memoryAllocated‚8.716812133789062) gpu.0.memoryAllocatedBytes‚ 7487684608 gpu.0.temp‚45.0 gpu.0.powerWatts‚110.871' gpu.0.enforcedPowerLimitWatts‚300.0 gpu.0.powerPercent‚36.957 gpu.0.smClock‚1410 gpu.0.memoryClock‚1512! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÀË22¾ - _wandbruntime‚4938  _runtime‚4938ÈYÈ¿  _step‚975 & lr-AdamW/pg1‚9.089911731158799e-05 & lr-AdamW/pg2‚9.089911731158799e-05 trainer/global_step‚12049 % _timestamp‚1.7845067848159525e+09 _runtime‚4938.879115896ãõçÃÍÀ Ç & lr-AdamW/pg1‚9.089911731158799e-05 & lr-AdamW/pg2‚9.089911731158799e-05 trainer/global_step‚12049 % _timestamp‚1.7845067848159525e+09 _runtime‚4938.879115896 _step‚975Ïu›ô2Á - _wandbruntime‚4938  _runtime‚4938²³P€< ×O HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚473349758558208 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84086575 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚458845005011632 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚473349653496832 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚467304195187582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚464864097297408 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 214007346 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8485661260800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚467304091999748 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚464864097297408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 % _timestamp‚1.7845067848169565e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91784059 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91784057 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8459173136590 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122223289 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚464864080258048 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91784057 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚464864080258048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8734179421184 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚473349653496832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8459086988116 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚452758503259136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214017491 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8485573238784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8733999007744 trainer/global_step‚12049 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚444024504251392$ ûâ'à ÜO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122233432 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚473349758558208 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 103187834 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 6618611712 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 318767104 _runtime‚4938.880003088 _step‚976 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚12374 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚10145 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91784059 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91784057 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8459173136590 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 318767104 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122223289 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚464864080258048 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚330 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91784057 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚464864080258048 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚24903680 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8734179421184 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 311682751 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚473349653496832 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8459086988116 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 6159996928 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 5878632448 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚452758503259136 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 771838464 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 6937378816 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 105061376 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚88022016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 214017491 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚482 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚88022016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8485573238784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8733999007744 trainer/global_step‚12049 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚10145 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 5878632448 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚444024504251392 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚152 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚12303 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 105061376 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚473349758558208 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84086575 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚458845005011632 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚473349653496832 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚12374 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 743477248 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚467304195187582 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 6937378816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚482 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚482 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚1362 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚464864097297408 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚482 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 6094181455 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚152 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 214007346 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8485661260800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚467304091999748 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 6618611712 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚464864097297408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 313566208 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚1362 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚3 % _timestamp‚1.7845067848169565e+09 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 5805303808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 214024565 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 105068544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 313566208 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚12374 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53112177 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚473387697973760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚458881894118752 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8486339377664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30981329 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚977 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 214024565 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122233187 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84093265 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚444060224312320 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91791380 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚452794898777600 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8486339377664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚464901463664640 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 trainer/global_step‚12049 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214034724 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91791378 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚444060199408640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8734699368960 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚464901446625280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8459849088146 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8459762939560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚467341743206898 _runtime‚4939.164526687 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214034724 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8486251348480 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚464901463664640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91791378 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8486251348480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚464901446625280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30981091 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84093506 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚452795108281856 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 103187946 % _timestamp‚1.7845067851011245e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 313566208 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53112174 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚86148586 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚473387803042304 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 5878632448 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91791380 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8734883969536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122233187 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 311682751 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚241%š‰AÆ æM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚473387697973760 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚238 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚458881894118752 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 6937378816 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8486339377664 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30981329 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 771838464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚482 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚330 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚152 _step‚977 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚88029184 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 214024565 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚482 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚10157 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122233187 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84093265 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚444060224312320 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚482 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚330 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91791380 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 6159996928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚12303 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚88029184 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚452794898777600 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 6094181455 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚330 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8486339377664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚482 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚464901463664640 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 6159996928 trainer/global_step‚12049 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 743477248 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 214034724 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91791378 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚3 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚482 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚444060199408640 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 184600576 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8734699368960 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚464901446625280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚1362 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚10157 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 201411584 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8459849088146 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8459762939560 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 318767104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚238 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚467341743206898 _runtime‚4939.164526687 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 214034724 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8486251348480 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 6937378816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚152 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚12374 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚464901463664640 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚717 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91791378 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8486251348480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚464901446625280 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30981091 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 6618611712 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 6618611712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84093506 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_bat